# Does math.stackexchange provide any tool for users to download questions in bulk.

Few years ago, when I did not have an account on this website, I used to note question id in my notebook for future reference to the question( I did not have a personal computer so there was no option to bookmark things and all, and also I did not know about the fact that I can save bookmarks online etc)

Now it is too tiring for me to type question id each time I want to access those question.

Is there an API kind of thing(as Facebook and Twitter have) using which I can download the question I want, by writing a small code or something like that?

I mean does math.stackexchange provide any tool for users to download questions in bulk.

I am sorry if my question doesn't make any sense. Thank you

Edit: By question id, I mean the unique number associated with each question in the URL

eg: 3155081

will correspond to How to find a counterexample for the following problem

# Choice of technologies

Since SE API gets real-time data, while SEDE returns data several days ago, the former will be used.

It's better to write a web solution with AJAX, but since I'm not a web developer (even though I maintain the public GitLab instance of Staticman for comments support on static sites), I'll use command line utilities in Unix shell as well as Python's json module (ships with Python).

# Fetch multiple users' favorite questions

I'll use OP's and my favorite questions as an example.

USERS="581242;290189"
curl "https://api.stackexchange.com/2.2/users/$USERS/favorites?order=desc&sort=activity&site=math" | \ gzip -d | python -m json.tool > temp.json awk 'BEGIN { print " QID | Title " print " ------ | ---------" ORS = "" } /question_id/ && match($$2, /[0-9]+/) { print sprintf("%7i", substr(2, RSTART, RLENGTH)) " | " } match($$0, "title\": \".*") { print substr($0, RSTART+9, RLENGTH-11) "\n"
}
' temp.json


Explanation of the above commands:

1. Define a list of USERS separated by ;
• at most 100 users is allowed here since that's the limit of {ids} in the API method below
• in case of multiple users, wrap the input with a pair of "'s, so that the shell would not interpret the last user ID (UID) as a shell command
• no whitespaces are allowed around =
2. Send the user list to SE API's /users/{ids}/favorites method with {ids} set to USERS so as to get a gzipped list of questions starred by any user in the list.
• The URL is too long, so I break it with \.
• The API's response is compressed with Gzip, so don't directly display the output to avoid messing up your shell.
• The results are sorted by activity in descending order in terms of timestamp.
• Only the first 30 questions are returned. See "paging" below.
3. Pipe the downloaded response to gzip -d for decompression. You would get a single-line raw JSON without trailing newline \n.
4. Pretty print the raw JSON using the command line interface of Python's json module, and redirect the standard output (STDOUT) to the file temp.json.
• Make sure that temp.json doesn't exist before running this command because it'll be overwritten by the output of the Python's JSON tool.
• I prefer saving it into a file instead of pipes so that you may reuse (other parts of) the data in the future.
5. The key extraction step: AWK

Basically, an AWK program is a (sequence of) pattern {action} running on each record. By default, records are delimited by newlines \n. (RS: record separator)

pattern can be

The most basic {action} is {print}, which print the current record to STDOUT. Each record has different fields, separated by by default. (FS: field separator)

• {print $1} prints the 1st column; • {print$2} prints the 2nd column; and so on
• {print $NF} prints the last column; (NF: number of fields) • {print$0} prints the whole record.

In this section, I keep the default RS (\n), so each line is treated as a record. What my AWK script does:

2. Set ORS (output record separator) to empty "".
• Since in temp.json, question_id and title are put into different lines, different patterns and {action}s are needed. The output of different {action}s are treated as different records. However, in the table, QID and title are in the same line, so ORS can't be the default newline \n.
• I tried setting ORS = " ", but the left alignment of the first line under the table header didn't match that of the rest.
3. Find the line matching question_id and print only the QID and the output table column border |.
• $2 means the 2nd "column" (delimited by by default) • match($0, /pat) {print substr($0,RSTART,RLENGTH)} returns only the matched string Note that this works regardless of leading whitespaces on the left-hand side since RSTART is a relative index. • && means AND in logic. • sprintf("%7i", string) returns a string formatted as an integer taking at most seven characters in width. 4. Perform a similar match for title. • Note the shifts +9 and -11. • This works since colons : in post title will be escaped by backslash \. • Print newline '\n' at the end to start a new output table row. 5. Paging (optional) • You may add the parameter pagesize ranging between 0 and 100 and it defaults to 30 https://api.stackexchange.com/2.2/users/$USERS/favorites?pagesize=100&order=desc&sort=activity&site=math

• You may use page to switch between pages. It defaults to 1. It's possible to write a while/for-loop to grab the entire list of favorite questions, but I left that as an exercise.

Output (first 30 questions )

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
100  5928  100  5928    0     0  12532      0 --:--:-- --:--:-- --:--:-- 12506
QID   |   Title
------ | ---------
3210162 | Minimal polynomial of extension of degree 2 over a finite field with characteristic 2
3210220 | Prove that there is no group $G$ s.t. $\\operatorname{Aut}(G)=\\mathbb{Q}$
3187120 | O is a point in triangle ABC. OA, OB and OC are joined and produced to meet BC, AC and AB at D, E and F. Find the value of OD/AD+OE/BE+OF/CF.
223851 | Parallelogram identity in the wave equation
222237 | Irreducible polynomial means no roots?
337937 | Why $\\sum_{k=1}^{\\infty} \\frac{k}{2^k} = 2$?
3206789 | $f : \\mathbb{R} \\to \\mathbb{R}$ be twice differentiable such that $f(0)=0, f(1/n)=0 \\forall n\\in \\mathbb{N}$ show that $f&#39;&#39;(0)=0$
3201343 | Book questions on supremum, bounds.
3202967 | Baby Rudin 2.17
1576885 | Let $f: \\Bbb R \\to [0,\\infty)$ be a continuous function such that $g(x)=(f(x))^2$ is uniformly continuous. Which of the following is always true?
477 | Cardinality of set of real continuous functions
30732 | How can I evaluate $\\sum_{n=0}^\\infty(n+1)x^n$?
2047606 | Comparing the rank of the product of two matrices with the ranks of the factors
1845542 | Irreducible polynomial over $\\mathbb{Q}$ implies polynomial is irreducible over $\\mathbb{Z}$
3 | List of interesting math podcasts?
3196648 | Can I say $1/x^2$ is not uniformly continuous on $[0, \\infty)$ because it is not defined when x = $0$?
3195512 | which of following option is correct for heat equation (CSIR june 2018)
3194826 | Are the endpoints of the domain of a function counted as critical points?
2049398 | How to plot a SDE $dx(t)=Ax(t)dt+Bx(t)dW(t)$ using Matlab, and how will the graph look like?
3193758 | If $AB=-BA$ then do $A$ and $B$ share a common eigenvector?
3193315 | Zeros of $f&#39;&#39;$
3193209 | Prove that $y_1=y_2$ or $\\{x \\in [a,b] \\mid y_1(x)=y_2(x)\\}$ is finite
3192961 | Suppose $f : [0, 1] \\to [0, 1]$ is differentiable such that $|f&#39;(x)|\\ne1$ for all $x\\in[0, 1]$. Prove $\\exists !a,b\\in[0,1]:f(a)=a,f(b)=1-b$
325149 | Number of socks to guarantee getting a matching pair.
3191747 | Proving $\\lim _{n\\to\\infty}\\frac{1}{ a_n} \\neq \\alpha$ given $\\lim _{n\\to\\infty}a_n = 0$ and...
3190717 | Regarding steady state solution of $u_{t}= c^2 u_{xx}$ with $u_{x}(0,t) = c_{1}$ and $u(L,t) = c_{1}$?
3189897 | Proof-Verification: $\\int_a^b \\left(\\frac{b-x}{b-a}\\right)^{2018}f(x)\\,{\\rm d}x \\leq \\frac{1}{2019}\\int_a^b f(x)\\,{\\rm d}x$
3190601 | Limit as $x \\to 0$ of $\\big(x \\int_x^1{\\frac{f(t)}{\\sin^2(t)}}\\: dt\\big)$
3190518 | Find $\\lim_{ n \\to \\infty} \\frac{e^{-n}}{\\sqrt{n}} \\sum_{k=0}^\\infty \\frac{\\sqrt{k+ n }}{k!} (n+a)^k$
3188674 | Maximum value of $|Z_1-Z_2|^2 +|Z_2-Z_3|^2+|Z_3-Z_1|^2$


Stack Printer prints SE questions into HTML files.

Suppose that the table obtained in the previous section is saved into a file temp.txt. To extract a list of QID, the first two lines (table header) are ignored by the pattern NR > 2, where NR means current number of record. We mainly focus on the first column (field) $1. This is the same has having a list of QID without post titles (, which are in the 2nd column $2 in test.txt). We will prepend and append strings using AWK in order to convert QID 2620409 to Stack Printer URL

http://www.stackprinter.com/export?question=2620409&service=math.stackexchange


awk '