7
$\begingroup$

Few years ago, when I did not have an account on this website, I used to note question id in my notebook for future reference to the question( I did not have a personal computer so there was no option to bookmark things and all, and also I did not know about the fact that I can save bookmarks online etc)

Now it is too tiring for me to type question id each time I want to access those question.

Is there an API kind of thing(as Facebook and Twitter have) using which I can download the question I want, by writing a small code or something like that?

I mean does math.stackexchange provide any tool for users to download questions in bulk.

I am sorry if my question doesn't make any sense. Thank you

Edit: By question id, I mean the unique number associated with each question in the URL

eg: 3155081

will correspond to How to find a counterexample for the following problem

$\endgroup$
9
  • $\begingroup$ Thanks a lot @YuiTo for editing the grammatical errors:) $\endgroup$ Mar 20, 2019 at 11:58
  • $\begingroup$ It's not me. It's Grammarly. (but I'm not a salesperson!) $\endgroup$ Mar 20, 2019 at 12:04
  • $\begingroup$ By downloading questions in bulk you mean that you have list of ids and you want to download all of the corresponding questions? Or do you mean something like downloading all questions from a given day, or in a given tag, or some other criteria? $\endgroup$ Mar 20, 2019 at 12:46
  • $\begingroup$ I have a list of question ids and I want to download the corresponding questions. @MartinSleziak $\endgroup$ Mar 20, 2019 at 12:48
  • 3
    $\begingroup$ Not exactly the same, but perhaps from these posts some inspiration can be obtained. (I hope that somebody will suggest a better solution.) What is the best way to download a list of stack exchange questions for offline browsing? and How do I backup my Stackexchange favorites for offline use? $\endgroup$ Mar 20, 2019 at 12:53
  • $\begingroup$ @MartinSleziak Thanks $\endgroup$ Mar 20, 2019 at 12:54
  • $\begingroup$ Related (on Meta Stackexchange): Where are the Stack Exchange data dumps? and its linked/related posts. $\endgroup$
    – hardmath
    Mar 20, 2019 at 22:50
  • 2
    $\begingroup$ When you have time you can stop by in the room dedicated to Addons and Scripts. I have at least mentioned there some partial solutions I am able to suggest: chat.stackexchange.com/rooms/90615/conversation/… Maybe we might be able to figure out something which might work for you with the tools you have at your disposal. $\endgroup$ Mar 22, 2019 at 12:46
  • $\begingroup$ Thanks Martin, I really appreciate that :) @MartinSleziak $\endgroup$ Mar 22, 2019 at 13:05

1 Answer 1

2
$\begingroup$

Choice of technologies

Since SE API gets real-time data, while SEDE returns data several days ago, the former will be used.

It's better to write a web solution with AJAX, but since I'm not a web developer (even though I maintain the public GitLab instance of Staticman for comments support on static sites), I'll use command line utilities in Unix shell as well as Python's json module (ships with Python).

Fetch multiple users' favorite questions

I'll use OP's and my favorite questions as an example.

USERS="581242;290189"
curl "https://api.stackexchange.com/2.2/users/$USERS/favorites?order=desc&sort=activity&site=math" | \
gzip -d | python -m json.tool > temp.json

awk 'BEGIN {
  print "  QID   |   Title   "
  print " ------ | ---------"
  ORS = ""
}
/question_id/ && match($2, /[0-9]+/) {
  print sprintf("%7i", substr($2, RSTART, RLENGTH)) " | "
}
match($0, "title\": \".*") {
  print substr($0, RSTART+9, RLENGTH-11) "\n"
}
' temp.json

Explanation of the above commands:

  1. Define a list of USERS separated by ;
    • at most 100 users is allowed here since that's the limit of {ids} in the API method below
    • in case of multiple users, wrap the input with a pair of "'s, so that the shell would not interpret the last user ID (UID) as a shell command
    • no whitespaces are allowed around =
  2. Send the user list to SE API's /users/{ids}/favorites method with {ids} set to USERS so as to get a gzipped list of questions starred by any user in the list.
    • The URL is too long, so I break it with \.
    • The API's response is compressed with Gzip, so don't directly display the output to avoid messing up your shell.
    • The results are sorted by activity in descending order in terms of timestamp.
    • Only the first 30 questions are returned. See "paging" below.
  3. Pipe the downloaded response to gzip -d for decompression. You would get a single-line raw JSON without trailing newline \n.
  4. Pretty print the raw JSON using the command line interface of Python's json module, and redirect the standard output (STDOUT) to the file temp.json.
    • Make sure that temp.json doesn't exist before running this command because it'll be overwritten by the output of the Python's JSON tool.
    • I prefer saving it into a file instead of pipes so that you may reuse (other parts of) the data in the future.
  5. The key extraction step: AWK

    Basically, an AWK program is a (sequence of) pattern {action} running on each record. By default, records are delimited by newlines \n. (RS: record separator)

    pattern can be

    • /regex/ (as in sed);
    • /start/,/end/ (as in sed); or
    • omitted (applies to each record).

    The most basic {action} is {print}, which print the current record to STDOUT. Each record has different fields, separated by by default. (FS: field separator)

    • {print $1} prints the 1st column;
    • {print $2} prints the 2nd column; and so on
    • {print $NF} prints the last column; (NF: number of fields)
    • {print $0} prints the whole record.

    In this section, I keep the default RS (\n), so each line is treated as a record. What my AWK script does:

    1. Print table header.
    2. Set ORS (output record separator) to empty "".
      • Since in temp.json, question_id and title are put into different lines, different patterns and {action}s are needed. The output of different {action}s are treated as different records. However, in the table, QID and title are in the same line, so ORS can't be the default newline \n.
      • I tried setting ORS = " ", but the left alignment of the first line under the table header didn't match that of the rest.
    3. Find the line matching question_id and print only the QID and the output table column border |.
      • $2 means the 2nd "column" (delimited by by default)
      • match($0, /pat) {print substr($0,RSTART,RLENGTH)} returns only the matched string Note that this works regardless of leading whitespaces on the left-hand side since RSTART is a relative index.
      • && means AND in logic.
      • sprintf("%7i", string) returns a string formatted as an integer taking at most seven characters in width.
    4. Perform a similar match for title.
      • Note the shifts +9 and -11.
      • This works since colons : in post title will be escaped by backslash \.
      • Print newline '\n' at the end to start a new output table row.
    5. Paging (optional)

      • You may add the parameter pagesize ranging between 0 and 100 and it defaults to 30

        https://api.stackexchange.com/2.2/users/$USERS/favorites?pagesize=100&order=desc&sort=activity&site=math
        
      • You may use page to switch between pages. It defaults to 1. It's possible to write a while/for-loop to grab the entire list of favorite questions, but I left that as an exercise.

Output (first 30 questions )

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5928  100  5928    0     0  12532      0 --:--:-- --:--:-- --:--:-- 12506
  QID   |   Title   
 ------ | ---------
3210162 | Minimal polynomial of extension of degree 2 over a finite field with characteristic 2
3210220 | Prove that there is no group $G$ s.t. $\\operatorname{Aut}(G)=\\mathbb{Q}$
3187120 | O is a point in triangle ABC. OA, OB and OC are joined and produced to meet BC, AC and AB at D, E and F. Find the value of OD/AD+OE/BE+OF/CF.
 223851 | Parallelogram identity in the wave equation
 222237 | Irreducible polynomial means no roots?
 337937 | Why $\\sum_{k=1}^{\\infty} \\frac{k}{2^k} = 2$?
3206789 | $f : \\mathbb{R} \\to \\mathbb{R}$ be twice differentiable such that $f(0)=0, f(1/n)=0 \\forall n\\in \\mathbb{N}$ show that $f''(0)=0$
3201343 | Book questions on supremum, bounds.
3202967 | Baby Rudin 2.17
1576885 | Let $f: \\Bbb R \\to [0,\\infty)$ be a continuous function such that $g(x)=(f(x))^2$ is uniformly continuous. Which of the following is always true?
    477 | Cardinality of set of real continuous functions
  30732 | How can I evaluate $\\sum_{n=0}^\\infty(n+1)x^n$?
2047606 | Comparing the rank of the product of two matrices with the ranks of the factors
1845542 | Irreducible polynomial over $\\mathbb{Q}$ implies polynomial is irreducible over $\\mathbb{Z}$
      3 | List of interesting math podcasts?
3196648 | Can I say $1/x^2$ is not uniformly continuous on $[0, \\infty)$ because it is not defined when x = $0$?
3195512 | which of following option is correct for heat equation (CSIR june 2018)
3194826 | Are the endpoints of the domain of a function counted as critical points?
2049398 | How to plot a SDE $dx(t)=Ax(t)dt+Bx(t)dW(t) $ using Matlab, and how will the graph look like?
3193758 | If $AB=-BA$ then do $A$ and $B$ share a common eigenvector?
3193315 | Zeros of $ f''$
3193209 | Prove that $y_1=y_2$ or $\\{x \\in [a,b] \\mid y_1(x)=y_2(x)\\}$ is finite
3192961 | Suppose $f : [0, 1] \\to [0, 1]$ is differentiable such that $|f'(x)|\\ne1$ for all $x\\in[0, 1]$. Prove $\\exists !a,b\\in[0,1]:f(a)=a,f(b)=1-b$
 325149 | Number of socks to guarantee getting a matching pair.
3191747 | Proving $\\lim _{n\\to\\infty}\\frac{1}{ a_n} \\neq \\alpha$ given $\\lim _{n\\to\\infty}a_n = 0$ and...
3190717 | Regarding steady state solution of $u_{t}= c^2 u_{xx}$ with $u_{x}(0,t) = c_{1}$ and $u(L,t) = c_{1}$?
3189897 | Proof-Verification: $ \\int_a^b \\left(\\frac{b-x}{b-a}\\right)^{2018}f(x)\\,{\\rm d}x \\leq \\frac{1}{2019}\\int_a^b f(x)\\,{\\rm d}x$
3190601 | Limit as $x \\to 0$ of $\\big(x \\int_x^1{\\frac{f(t)}{\\sin^2(t)}}\\: dt\\big)$
3190518 | Find $\\lim_{ n \\to \\infty} \\frac{e^{-n}}{\\sqrt{n}} \\sum_{k=0}^\\infty \\frac{\\sqrt{k+ n }}{k!} (n+a)^k$
3188674 | Maximum value of $|Z_1-Z_2|^2 +|Z_2-Z_3|^2+|Z_3-Z_1|^2$

Batch download from Stack Printer

Stack Printer prints SE questions into HTML files.

Suppose that the table obtained in the previous section is saved into a file temp.txt. To extract a list of QID, the first two lines (table header) are ignored by the pattern NR > 2, where NR means current number of record. We mainly focus on the first column (field) $1. This is the same has having a list of QID without post titles (, which are in the 2nd column $2 in test.txt). We will prepend and append strings using AWK in order to convert QID 2620409 to Stack Printer URL

http://www.stackprinter.com/export?question=2620409&service=math.stackexchange

Here's my actual command for batch downloading.

awk '              
BEGIN {
  baseURL="http://www.stackprinter.com/export?question="
  param="\&service=math.stackexchange"
} 
NR > 2 {
  system("wget " baseURL $1 param " -O " $1".html && sleep $(( ( RANDOM % 3 ) + 1 ))")
}' temp.txt

As usual, the constants are set in BEGIN {...}. This is not necessary, but I prefer setting them at the beginning so that the code can be easily adapted and reused later.

param represents the parameters for this HTTP GET request. I've found out that the leading ampersand & in param needs to be escaped by trial and error. Perhaps it's because ampersand & makes jobs run in background in Unix shell.

The download action is executed from the third line. The shell command is supposed to be

wget http://www.stackprinter.com/export?question={qid}&service=math.stackexchange -O {qid}.html

The utility wget downloads the SE question (specified by paramtersquestion and service) to a file. Without the option -O, the saved file name would default to the entire substring after the last slash / in the request URL. Since filenames like

ls
...
'export?question=1576885&service=math.stackexchange'  'export?question=3&service=math.stackexchange'
'export?question=1845542&service=math.stackexchange'  'export?question=477&service=math.stackexchange'
...

are undesirable, we need something like -O {qid}.html so that the GUI file explorer will recognise that it's an HTTP document to be opened in a web browser.

The above wget command has to be adapted and called inside the AWK script by the action { system(...) }. This is as simple as wrapping the wget command by double quotes " and leaving AWK's variables outside the "'s. Note that the double-quoted spaces control the spaces in the wget command called, whereas the un-double-quoted spaces don't matter.

Last but not least, I've introduced a brief random pause lasting for 1/2/3 second(s) for friendly crawl. After having written the above code block, I realized there's no need to reinvent the wheel. wget has already provided options --wait=10 --random-wait and --input-file=url.txt. I'm leaving these as an exercise.

Remarks: Failed sed + xargs way

I don't know a simpler way to get it done by sed + xargs.

sed -ne '3,$s/^\s*//p' temp.txt | cut -d' ' -f1 | \
sed -e 's/^/https:\/\/www.stackprinter.com\/export?question=/' \
-e 's/$/\&service=math.stackexchange/' | xargs wget ???

The sed way gets complicated when you need to repeat the column {qid} twice for specifying wget's output file name with wget {baseURL}{qid}{param} -O {qid}.html. Once you know awk's intuitive { print "foo " $3 " bar " $1, $3 }, you won't want to spend your time escaping the slashes / in sed's s command.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .