Choice of technologies
Since SE API gets real-time data, while SEDE returns data several days ago, the former will be used.
It's better to write a web solution with AJAX, but since I'm not a web developer (even though I maintain the public GitLab instance of Staticman for comments support on static sites), I'll use command line utilities in Unix shell as well as Python's json module (ships with Python).
Fetch multiple users' favorite questions
I'll use OP's and my favorite questions as an example.
USERS="581242;290189"
curl "https://api.stackexchange.com/2.2/users/$USERS/favorites?order=desc&sort=activity&site=math" | \
gzip -d | python -m json.tool > temp.json
awk 'BEGIN {
print " QID | Title "
print " ------ | ---------"
ORS = ""
}
/question_id/ && match($2, /[0-9]+/) {
print sprintf("%7i", substr($2, RSTART, RLENGTH)) " | "
}
match($0, "title\": \".*") {
print substr($0, RSTART+9, RLENGTH-11) "\n"
}
' temp.json
Explanation of the above commands:
- Define a list of
USERS separated by ;
- at most 100 users is allowed here since that's the limit of
{ids} in the API method below
- in case of multiple users, wrap the input with a pair of
"'s, so that the shell would not interpret the last user ID (UID) as a shell command
- no whitespaces
=
- Send the user list to SE API's
/users/{ids}/favorites method with {ids} set to USERS so as to get a gzipped list of questions starred by any user in the list.
- The URL is too long, so I break it with
\.
- The API's response is compressed with Gzip, so don't directly display the output to avoid messing up your shell.
- The results are sorted by activity in descending order in terms of timestamp.
- Only the first 30 questions are returned. See "paging" below.
- Pipe the downloaded response to
gzip -d for decompression. You would get a single-line raw JSON without trailing newline \n.
- Pretty print the raw JSON using the command line interface of Python's
json module, and redirect the standard output (STDOUT) to the file temp.json.
- Make sure that
temp.json doesn't exist before running this command because it'll be overwritten by the output of the Python's JSON tool.
- I prefer saving it into a file instead of pipes so that you may reuse (other parts of) the data in the future.
The key extraction step: AWK
Basically, an AWK program is a (sequence of) pattern {action} running on each record. By default, records are delimited by newlines \n. (RS: record separator)
pattern can be
/regex/ (as in sed);/start/,/end/ (as in sed); or- omitted (applies to each record).
The most basic {action} is {print}, which print the current record to STDOUT.
Each record has different fields, separated by FS: field separator)
{print $1} prints the 1st column;{print $2} prints the 2nd column; and so on{print $NF} prints the last column; (NF: number of fields){print $0} prints the whole record.
In this section, I keep the default RS (\n), so each line is treated as a record. What my AWK script does:
- Print table header.
- Set
ORS (output record separator) to empty "".
- Since in
temp.json, question_id and title are put into different lines, different patterns and {action}s are needed. The output of different {action}s are treated as different records. However, in the table, QID and title are in the same line, so ORS can't be the default newline \n.
- I tried setting
ORS = " ", but the left alignment of the first line under the table header didn't match that of the rest.
- Find the line matching
question_id and print only the QID and the output table column border |.
$2 means the 2nd "column" (delimited by match($0, /pat) {print substr($0,RSTART,RLENGTH)} returns only the matched string Note that this works regardless of leading whitespaces RSTART is a relative index.&& means AND in logic.sprintf("%7i", string) returns a string formatted as an integer taking at most seven characters in width.
- Perform a similar match for
title.
- Note the shifts
+9 and -11.
- This works since colons
: in post title will be escaped by backslash \.
- Print newline '\n' at the end to start a new output table row.
Paging (optional)
You may add the parameter pagesize ranging between 0 and 100 and it defaults to 30
https://api.stackexchange.com/2.2/users/$USERS/favorites?pagesize=100&order=desc&sort=activity&site=math
You may use page to switch between pages. It defaults to 1. It's possible to write a while/for-loop to grab the entire list of favorite questions, but I left that as an exercise.
Output (first 30 questions )
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5928 100 5928 0 0 12532 0 --:--:-- --:--:-- --:--:-- 12506
QID | Title
------ | ---------
3210162 | Minimal polynomial of extension of degree 2 over a finite field with characteristic 2
3210220 | Prove that there is no group $G$ s.t. $\\operatorname{Aut}(G)=\\mathbb{Q}$
3187120 | O is a point in triangle ABC. OA, OB and OC are joined and produced to meet BC, AC and AB at D, E and F. Find the value of OD/AD+OE/BE+OF/CF.
223851 | Parallelogram identity in the wave equation
222237 | Irreducible polynomial means no roots?
337937 | Why $\\sum_{k=1}^{\\infty} \\frac{k}{2^k} = 2$?
3206789 | $f : \\mathbb{R} \\to \\mathbb{R}$ be twice differentiable such that $f(0)=0, f(1/n)=0 \\forall n\\in \\mathbb{N}$ show that $f''(0)=0$
3201343 | Book questions on supremum, bounds.
3202967 | Baby Rudin 2.17
1576885 | Let $f: \\Bbb R \\to [0,\\infty)$ be a continuous function such that $g(x)=(f(x))^2$ is uniformly continuous. Which of the following is always true?
477 | Cardinality of set of real continuous functions
30732 | How can I evaluate $\\sum_{n=0}^\\infty(n+1)x^n$?
2047606 | Comparing the rank of the product of two matrices with the ranks of the factors
1845542 | Irreducible polynomial over $\\mathbb{Q}$ implies polynomial is irreducible over $\\mathbb{Z}$
3 | List of interesting math podcasts?
3196648 | Can I say $1/x^2$ is not uniformly continuous on $[0, \\infty)$ because it is not defined when x = $0$?
3195512 | which of following option is correct for heat equation (CSIR june 2018)
3194826 | Are the endpoints of the domain of a function counted as critical points?
2049398 | How to plot a SDE $dx(t)=Ax(t)dt+Bx(t)dW(t) $ using Matlab, and how will the graph look like?
3193758 | If $AB=-BA$ then do $A$ and $B$ share a common eigenvector?
3193315 | Zeros of $ f''$
3193209 | Prove that $y_1=y_2$ or $\\{x \\in [a,b] \\mid y_1(x)=y_2(x)\\}$ is finite
3192961 | Suppose $f : [0, 1] \\to [0, 1]$ is differentiable such that $|f'(x)|\\ne1$ for all $x\\in[0, 1]$. Prove $\\exists !a,b\\in[0,1]:f(a)=a,f(b)=1-b$
325149 | Number of socks to guarantee getting a matching pair.
3191747 | Proving $\\lim _{n\\to\\infty}\\frac{1}{ a_n} \\neq \\alpha$ given $\\lim _{n\\to\\infty}a_n = 0$ and...
3190717 | Regarding steady state solution of $u_{t}= c^2 u_{xx}$ with $u_{x}(0,t) = c_{1}$ and $u(L,t) = c_{1}$?
3189897 | Proof-Verification: $ \\int_a^b \\left(\\frac{b-x}{b-a}\\right)^{2018}f(x)\\,{\\rm d}x \\leq \\frac{1}{2019}\\int_a^b f(x)\\,{\\rm d}x$
3190601 | Limit as $x \\to 0$ of $\\big(x \\int_x^1{\\frac{f(t)}{\\sin^2(t)}}\\: dt\\big)$
3190518 | Find $\\lim_{ n \\to \\infty} \\frac{e^{-n}}{\\sqrt{n}} \\sum_{k=0}^\\infty \\frac{\\sqrt{k+ n }}{k!} (n+a)^k$
3188674 | Maximum value of $|Z_1-Z_2|^2 +|Z_2-Z_3|^2+|Z_3-Z_1|^2$
Batch download from Stack Printer
Stack Printer prints SE questions into HTML files.
Suppose that the table obtained in the previous section is saved into a file temp.txt. To extract a list of QID, the first two lines (table header) are ignored by the pattern NR > 2, where NR means current number of record. We mainly focus on the first column (field) $1. This is the same has having a list of QID without post titles (, which are in the 2nd column $2 in test.txt). We will prepend and append strings using AWK in order to convert QID 2620409 to Stack Printer URL
http://www.stackprinter.com/export?question=2620409&service=math.stackexchange
Here's my actual command for batch downloading.
awk '
BEGIN {
baseURL="http://www.stackprinter.com/export?question="
param="\&service=math.stackexchange"
}
NR > 2 {
system("wget " baseURL $1 param " -O " $1".html && sleep $(( ( RANDOM % 3 ) + 1 ))")
}' temp.txt
As usual, the constants are set in BEGIN {...}. This is not necessary, but I prefer setting them at the beginning so that the code can be easily adapted and reused later.
param represents the parameters for this HTTP GET request. I've found out that the leading ampersand & in param needs to be escaped by trial and error. Perhaps it's because ampersand & makes jobs run in background in Unix shell.
The download action is executed from the third line. The shell command is supposed to be
wget http://www.stackprinter.com/export?question={qid}&service=math.stackexchange -O {qid}.html
The utility wget downloads the SE question (specified by paramtersquestion and service) to a file. Without the option -O, the saved file name would default to the entire substring after the last slash / in the request URL. Since filenames like
ls
...
'export?question=1576885&service=math.stackexchange' 'export?question=3&service=math.stackexchange'
'export?question=1845542&service=math.stackexchange' 'export?question=477&service=math.stackexchange'
...
are undesirable, we need something like -O {qid}.html so that the GUI file explorer will recognise that it's an HTTP document to be opened in a web browser.
The above wget command has to be adapted and called inside the AWK script by the action { system(...) }. This is as simple as wrapping the wget command by double quotes " and leaving AWK's variables outside the "'s. Note that the double-quoted spaces wget command called, whereas the un-double-quoted spaces
Last but not least, I've introduced a brief random pause lasting for 1/2/3 second(s) for friendly crawl. After having written the above code block, I realized there's no need to reinvent the wheel. wget has already provided options --wait=10 --random-wait and --input-file=url.txt. I'm leaving these as an exercise.
Remarks: Failed sed + xargs way
I don't know a simpler way to get it done by sed + xargs.
sed -ne '3,$s/^\s*//p' temp.txt | cut -d' ' -f1 | \
sed -e 's/^/https:\/\/www.stackprinter.com\/export?question=/' \
-e 's/$/\&service=math.stackexchange/' | xargs wget ???
The sed way gets complicated when you need to repeat the column {qid} twice for specifying wget's output file name with wget {baseURL}{qid}{param} -O {qid}.html. Once you know awk's intuitive { print "foo " $3 " bar " $1, $3 }, you won't want to spend your time escaping the slashes / in sed's s command.
