A guide to extracting data from Medium API

Does Medium have an API? This is a question that has been asked several times by users. The answer is yes. Medium's API is a JSON-based OAuth2 API. All requests are made to endpoints beginning: https://api.medium.com/v1 and all requests must be secure, i.e. https, not http.

We can interact with the Medium Api using the programming language of our choice solely to get the data we are interested in. In this post, we will use Python language to get our data. Here is the list of the data we would be getting.

  • List of publications by user
  • Statistical analysis of Medium account

First Things First: Integration Token

The first thing we need to do before we can start pulling data from Medium is to acquire an integration token. We can do this by going to settings under our profile tab and then click on integration token. Click on "get integration token" after passing a description for the token. This brings out a long list of hex strings that we can use as access tokens in our code.

We can also use the Developer option right below the integration token whereby we can create a new application here and then a client id and client secret will be given to us. We can as well use these ids' in our code to generate our access token.

Key setup steps

1. Go to Settings under your Medium profile tab

2. Click on Integration Token and generate one

3. Alternatively, use the Developer option to create an app and get a client_id and client_secret

Access Token

We will use the medium token Url endpoint method along with our credentials to request for an access token which we can pass to the user publication endpoint to get our data.

Python
#W will import our libraries here and copy our credentials from medium then put it in a python file(config.py) import requests import json from config import ClientID, ClientSecret, Code, Redirect import pandas as pd #We will write a function that uses the request library and the post method # along with the medium token url endpoint to send our credentials,so # we could get the access token we need. def mediumToken(): the_url = 'https://api.medium.com/v1/tokens' response = requests.post(the_url, headers = { 'Content-Type': 'application/x-www-form-urlencoded', 'Accept': 'application/json', 'Accept-Charset': 'utf-8' }, data = { "code": 'xxxxxxxx', "client_id": ClientID, "client_secret": ClientSecret, 'grant_type' : 'authorization_code', 'redirect_uri': Redirect }) response_data = json.loads(response.text) return response_data

Refresh Token

We will generate a new access token using the refresh token which has a longer time than the earlier access token generated. By doing this, we will not keep running into error showing "access token has expired"

Python
#Get a new access token using the access token and refresh token generated earlier, #We use the same tokens url endpoint along with other credentials but with a new grant type #specified as refresh token to generate the new access token. def NewAccessToken(): the_url = 'https://api.medium.com/v1/tokens' response = requests.post(the_url, headers = { 'Content-Type': 'application/x-www-form-urlencoded', 'Accept': 'application/json', 'Accept-Charset': 'utf-8' }, data = { "refresh_token": 'xxxxxxxxxxxxxxxx', "client_id": ClientID, "client_secret": ClientSecret, 'grant_type' : 'refresh_token', }) json_response = response.json() value=json_response['access_token'] return value

Get User Publications

We can get our user publications with the function below by passing our headers which have the access token. With the request library, we will send a get request which will return the data we need. Afterward, we will clean this data.

Note

Medium data are usually in RSS Format.

Python
def publications(): response = requests.get('https://api.medium.com/v1/users/publications', headers={ 'Authorization': 'Bearer ' + NewAccessToken(), 'Content-Type': 'application/x-www-form-urlencoded' }) return response.text

Get Statistical Data

We can get statistical data for a medium account including both audience and story stats. This data contains monthly growth, number of followers, viewers, readers, fans, and so on.

We'll be making use of the medium stats Library developed on top Pypi. This library has the command line usage but we'll be making use of the python approach.

Steps to follow

  1. Sign in to Medium account
  2. Go to the browser's developer tools (either Chrome or any browser) and find the tab which holds cookies: (in chrome, go to Settings → privacy & security → cookies & other sites data → see all cookies & sites data)
  3. Scroll through the cookies until you find medium and copy the uid and sid.

These two things will be used with the library inside our code to generate our data. We will then proceed to our code.

Python
# We will import the library here and datetime library we'll need, #Then parse the sid and uid we copied here. #We will also use the datetime library to create a start and stop variable. from medium_stats.scraper import StatGrabberUser import datetime import pandas as pd sidd="1:gWiuIAhS1bVVr6nrBqtX4msYjpWVdvh/XYQkOfenra3zf/mAG48S4I3/lJZPUAX5" uidd="1a02447fe0ee" start_time = '1/01/20' start=datetime.datetime.strptime(start_time, '%m/%d/%y') stop = datetime.datetime.today() + datetime.timedelta(days=1)

We will define the function to parse all our parameters

Python
#We will define an empty list here to append all data coming from the function all_time_referrers=[] daily_event_logs=[] event_logs=[] summary_stats=[] articles_ids=[] def data(): # get aggregated summary statistics; note: start/stop will be converted to UTC #we then pass our username, the sid and uid, with the start and stop time me = StatGrabberUser('cndro', sid=sidd, uid=uidd, start=start, stop=stop) #we use the getsummarystatistics to get a summary of our monthly activities data = me.get_summary_stats() # get the unattributed event logs for all your stories: data_events = me.get_summary_stats(events=True) articles = me.get_article_ids(data) # returns a list of article_ids article_events = me.get_all_story_stats(articles) # daily event logs referrers = me.get_all_story_stats(articles, type_='referrer') # all-time referral sources summary=summary_stats.append(data) ids=articles_ids.append(articles) evt=event_logs.append(data_events) daily_evt=daily_event_logs.append(article_events) referr=all_time_referrers.append(referrers) return summary, ids, evt, daily_evt, referr

Data Cleaning

With the various data we saved in a list, we can demonstrate with the event logs data by cleaning it and see what the data looks like.

Let's open an empty dictionary here and loop through the events logs and parse an event key inside

Python
evt={} for i in event_logs: evt['events'] = i

We will calculate the length of the events using the earlier dictionary we created.

Python
the_size=len(evt['events'])

Now, we can create a different empty list to which we can append all data coming from our code

Python
userId=[] flaggedSpam=[] timestamp=[] upvotes=[] reads=[] views=[] claps=[] updateNotificationSubscribers=[]

We will loop through the data using the size of the data to get all the data we need and then append each to its related opened empty list.

Python
for i in range(0, the_size): userId.append(evt['events'][i]['userId']) flaggedSpam.append(evt['events'][i]['flaggedSpam']) timestamp.append((time.ctime(int(evt['events'][i]['timestampMs'])/1000))) upvotes.append(evt['events'][i]['upvotes']) reads.append(evt['events'][i]['reads']) views.append(evt['events'][i]['views']) claps.append(evt['events'][i]['claps']) updateNotificationSubscribers.append(evt['events'][i]['updateNotificationSubscribers'])

Finally, we will use Pandas to convert these lists we created and combine them all to one DataFrame.

Python
dataframes = {'userId': userId, 'flaggedSpam': flaggedSpam, 'timestamp': timestamp, 'upvotes':upvotes, 'reads':reads, 'views': views, 'claps': claps, 'updateNotificationSubscribers': updateNotificationSubscribers} df_events = pd.DataFrame(data=dataframes)

Conclusion

We have come to the end of this tutorial. I hope you will follow the steps above and go ahead to get data from Medium API successfully.

Originally published on Medium. Read the original article →
Need help with your data?

Let CNDRO build the data infrastructure your business needs

From API integrations to analytics pipelines — we engineer the backend so you can focus on growth.

Book a free consultation →