Choice of technologies
Since SE API gets real-time data, while SEDE returns data several days ago, the former will be used.
It's better to write a web solution with AJAX, but since I'm not a web developer (even though I maintain the public GitLab instance of Staticman for comments support on static sites), I'll use command line utilities in Unix shell as well as Python's json
module (ships with Python).
Fetch multiple users' favorite questions
I'll use OP's and my favorite questions as an example.
USERS="581242;290189"
curl "https://api.stackexchange.com/2.2/users/$USERS/favorites?order=desc&sort=activity&site=math" | \
gzip -d | python -m json.tool > temp.json
awk 'BEGIN {
print " QID | Title "
print " ------ | ---------"
ORS = ""
}
/question_id/ && match($2, /[0-9]+/) {
print sprintf("%7i", substr($2, RSTART, RLENGTH)) " | "
}
match($0, "title\": \".*") {
print substr($0, RSTART+9, RLENGTH-11) "\n"
}
' temp.json
Explanation of the above commands:
- Define a list of
USERS
separated by ;
- at most 100 users is allowed here since that's the limit of
{ids}
in the API method below
- in case of multiple users, wrap the input with a pair of
"
's, so that the shell would not interpret the last user ID (UID) as a shell command
- no whitespaces
are allowed around =
- Send the user list to SE API's
/users/{ids}/favorites
method with {ids}
set to USERS
so as to get a gzipped list of questions starred by any user in the list.
- The URL is too long, so I break it with
\
.
- The API's response is compressed with Gzip, so don't directly display the output to avoid messing up your shell.
- The results are sorted by activity in descending order in terms of timestamp.
- Only the first 30 questions are returned. See "paging" below.
- Pipe the downloaded response to
gzip -d
for decompression. You would get a single-line raw JSON without trailing newline \n
.
- Pretty print the raw JSON using the command line interface of Python's
json
module, and redirect the standard output (STDOUT) to the file temp.json
.
- Make sure that
temp.json
doesn't exist before running this command because it'll be overwritten by the output of the Python's JSON tool.
- I prefer saving it into a file instead of pipes so that you may reuse (other parts of) the data in the future.
The key extraction step: AWK
Basically, an AWK program is a (sequence of) pattern {action}
running on each record. By default, records are delimited by newlines \n
. (RS
: record separator)
pattern
can be
/regex/
(as in sed
);
/start/,/end/
(as in sed
); or
- omitted (applies to each record).
The most basic {action}
is {print}
, which print the current record to STDOUT.
Each record has different fields, separated by
by default. (FS
: field separator)
{print $1}
prints the 1st column;
{print $2}
prints the 2nd column; and so on
{print $NF}
prints the last column; (NF
: number of fields)
{print $0}
prints the whole record.
In this section, I keep the default RS
(\n
), so each line is treated as a record. What my AWK script does:
- Print table header.
- Set
ORS
(output record separator) to empty ""
.
- Since in
temp.json
, question_id
and title
are put into different lines, different pattern
s and {action}
s are needed. The output of different {action}
s are treated as different records. However, in the table, QID and title are in the same line, so ORS
can't be the default newline \n
.
- I tried setting
ORS = " "
, but the left alignment of the first line under the table header didn't match that of the rest.
- Find the line matching
question_id
and print only the QID and the output table column border |
.
$2
means the 2nd "column" (delimited by
by default)
match($0, /pat) {print substr($0,RSTART,RLENGTH)}
returns only the matched string Note that this works regardless of leading whitespaces
on the left-hand side since RSTART
is a relative index.
&&
means AND in logic.
sprintf("%7i", string)
returns a string formatted as an integer taking at most seven characters in width.
- Perform a similar match for
title
.
- Note the shifts
+9
and -11
.
- This works since colons
:
in post title will be escaped by backslash \
.
- Print newline '\n' at the end to start a new output table row.
Paging (optional)
You may add the parameter pagesize
ranging between 0
and 100
and it defaults to 30
https://api.stackexchange.com/2.2/users/$USERS/favorites?pagesize=100&order=desc&sort=activity&site=math
You may use page
to switch between pages. It defaults to 1
. It's possible to write a while
/for
-loop to grab the entire list of favorite questions, but I left that as an exercise.
Output (first 30 questions )
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5928 100 5928 0 0 12532 0 --:--:-- --:--:-- --:--:-- 12506
QID | Title
------ | ---------
3210162 | Minimal polynomial of extension of degree 2 over a finite field with characteristic 2
3210220 | Prove that there is no group $G$ s.t. $\\operatorname{Aut}(G)=\\mathbb{Q}$
3187120 | O is a point in triangle ABC. OA, OB and OC are joined and produced to meet BC, AC and AB at D, E and F. Find the value of OD/AD+OE/BE+OF/CF.
223851 | Parallelogram identity in the wave equation
222237 | Irreducible polynomial means no roots?
337937 | Why $\\sum_{k=1}^{\\infty} \\frac{k}{2^k} = 2$?
3206789 | $f : \\mathbb{R} \\to \\mathbb{R}$ be twice differentiable such that $f(0)=0, f(1/n)=0 \\forall n\\in \\mathbb{N}$ show that $f''(0)=0$
3201343 | Book questions on supremum, bounds.
3202967 | Baby Rudin 2.17
1576885 | Let $f: \\Bbb R \\to [0,\\infty)$ be a continuous function such that $g(x)=(f(x))^2$ is uniformly continuous. Which of the following is always true?
477 | Cardinality of set of real continuous functions
30732 | How can I evaluate $\\sum_{n=0}^\\infty(n+1)x^n$?
2047606 | Comparing the rank of the product of two matrices with the ranks of the factors
1845542 | Irreducible polynomial over $\\mathbb{Q}$ implies polynomial is irreducible over $\\mathbb{Z}$
3 | List of interesting math podcasts?
3196648 | Can I say $1/x^2$ is not uniformly continuous on $[0, \\infty)$ because it is not defined when x = $0$?
3195512 | which of following option is correct for heat equation (CSIR june 2018)
3194826 | Are the endpoints of the domain of a function counted as critical points?
2049398 | How to plot a SDE $dx(t)=Ax(t)dt+Bx(t)dW(t) $ using Matlab, and how will the graph look like?
3193758 | If $AB=-BA$ then do $A$ and $B$ share a common eigenvector?
3193315 | Zeros of $ f''$
3193209 | Prove that $y_1=y_2$ or $\\{x \\in [a,b] \\mid y_1(x)=y_2(x)\\}$ is finite
3192961 | Suppose $f : [0, 1] \\to [0, 1]$ is differentiable such that $|f'(x)|\\ne1$ for all $x\\in[0, 1]$. Prove $\\exists !a,b\\in[0,1]:f(a)=a,f(b)=1-b$
325149 | Number of socks to guarantee getting a matching pair.
3191747 | Proving $\\lim _{n\\to\\infty}\\frac{1}{ a_n} \\neq \\alpha$ given $\\lim _{n\\to\\infty}a_n = 0$ and...
3190717 | Regarding steady state solution of $u_{t}= c^2 u_{xx}$ with $u_{x}(0,t) = c_{1}$ and $u(L,t) = c_{1}$?
3189897 | Proof-Verification: $ \\int_a^b \\left(\\frac{b-x}{b-a}\\right)^{2018}f(x)\\,{\\rm d}x \\leq \\frac{1}{2019}\\int_a^b f(x)\\,{\\rm d}x$
3190601 | Limit as $x \\to 0$ of $\\big(x \\int_x^1{\\frac{f(t)}{\\sin^2(t)}}\\: dt\\big)$
3190518 | Find $\\lim_{ n \\to \\infty} \\frac{e^{-n}}{\\sqrt{n}} \\sum_{k=0}^\\infty \\frac{\\sqrt{k+ n }}{k!} (n+a)^k$
3188674 | Maximum value of $|Z_1-Z_2|^2 +|Z_2-Z_3|^2+|Z_3-Z_1|^2$
Batch download from Stack Printer
Stack Printer prints SE questions into HTML files.
Suppose that the table obtained in the previous section is saved into a file temp.txt
. To extract a list of QID, the first two lines (table header) are ignored by the pattern NR > 2
, where NR
means current number of record. We mainly focus on the first column (field) $1
. This is the same has having a list of QID without post titles (, which are in the 2nd column $2
in test.txt
). We will prepend and append strings using AWK in order to convert QID 2620409
to Stack Printer URL
http://www.stackprinter.com/export?question=2620409&service=math.stackexchange
Here's my actual command for batch downloading.
awk '
BEGIN {
baseURL="http://www.stackprinter.com/export?question="
param="\&service=math.stackexchange"
}
NR > 2 {
system("wget " baseURL $1 param " -O " $1".html && sleep $(( ( RANDOM % 3 ) + 1 ))")
}' temp.txt
As usual, the constants are set in BEGIN {...}
. This is not necessary, but I prefer setting them at the beginning so that the code can be easily adapted and reused later.
param
represents the parameters for this HTTP GET request. I've found out that the leading ampersand &
in param
needs to be escaped by trial and error. Perhaps it's because ampersand &
makes jobs run in background in Unix shell.
The download action is executed from the third line. The shell command is supposed to be
wget http://www.stackprinter.com/export?question={qid}&service=math.stackexchange -O {qid}.html
The utility wget
downloads the SE question (specified by paramtersquestion
and service
) to a file. Without the option -O
, the saved file name would default to the entire substring after the last slash /
in the request URL. Since filenames like
ls
...
'export?question=1576885&service=math.stackexchange' 'export?question=3&service=math.stackexchange'
'export?question=1845542&service=math.stackexchange' 'export?question=477&service=math.stackexchange'
...
are undesirable, we need something like -O {qid}.html
so that the GUI file explorer will recognise that it's an HTTP document to be opened in a web browser.
The above wget
command has to be adapted and called inside the AWK script by the action { system(...) }
. This is as simple as wrapping the wget
command by double quotes "
and leaving AWK's variables outside the "
's. Note that the double-quoted spaces
control the spaces in the wget
command called, whereas the un-double-quoted spaces
don't matter.
Last but not least, I've introduced a brief random pause lasting for 1/2/3 second(s) for friendly crawl. After having written the above code block, I realized there's no need to reinvent the wheel. wget
has already provided options --wait=10 --random-wait
and --input-file=url.txt
. I'm leaving these as an exercise.
Remarks: Failed sed
+ xargs
way
I don't know a simpler way to get it done by sed
+ xargs
.
sed -ne '3,$s/^\s*//p' temp.txt | cut -d' ' -f1 | \
sed -e 's/^/https:\/\/www.stackprinter.com\/export?question=/' \
-e 's/$/\&service=math.stackexchange/' | xargs wget ???
The sed
way gets complicated when you need to repeat the column {qid} twice for specifying wget
's output file name with wget {baseURL}{qid}{param} -O {qid}.html
. Once you know awk
's intuitive { print "foo " $3 " bar " $1, $3 }
, you won't want to spend your time escaping the slashes /
in sed
's s
command.