Part 1: Find and scrape the data with Selenium and Scrapy

identify the problem

For this project, I wanted to look at a specific feature to see if it could predict the performance of recently released movies. While many factors, such as critical acclaim, marketing, release season, and competition between films, can affect a film's performance in theaters, I wanted to try to answer one question:

In general, I expected to see a correlation between the opening weekend performance and the popularity of the cast. The hard part would be finding a way to quantify the "star power" aspect of a movie. To do this I had to find and scrape some data online!

Fearful! Now I have way more data than I need! Not that this is the worst, of course, but we have to filter a bit. To do this I decided to use Panda to import and clean the data.

title_basics_df = pd.read_csv('Data/title.basics.tsv', sep='\t')title_cast_df = pd.read_csv('Data/title.principals.tsv', sep='\t')title_ratings_df = PD. read_csv('Datos/título.calificaciones.tsv', sep='\t')

máscara = ((title_basics_df['startYear'] >= 2014) & (title_basics_df['startYear'] <= 2017) & (title_basics_df['titleType'] == 'film') & (title_basics_df']=A 0) & (title_basics_df['runtimeMinutes'] > 80) & (title_basics_df['genres'] != '') & (title_basics_df['genres'] != 'Documentaire'))

I also had to clean up the data before applying these filters. Some columns weren't needed and some weren't formatted correctly (such as runtime years and minutes), so I created some helper functions to clean them up:

## Helperfunctiesdef clean_year(y): # Retourneer jaar als een geheel getal of "NaN" indien leeg importer numpy als np probeer: return int(y) behalve: return np.nandef clean_genre(y): # Retourneer allemeldehetre y = str( y) si y == '\\N': return '' return y.split(',')[0].strip()title_basics_df.drop('endYear', as=1, inplace=True )title_basics_df[' startYear'] = title_basics_df['startYear'].apply(clean_year)title_basics_df['runtimeMinutes'] = title_basics_df['runtimeMinutes'].apply(clean_year)title['gen_resics]df'title_genres_df .apply(clean_genre)title_basics_df.dropna( inplace=True, how='any', subset=['startYear', 'runtimeMinutes'])

So we just need to merge the data,

títulos = title_basics_df[masker].merge(title_cast_df, on='tconst')titles = títulos.merge(title_ratings_df, on='tconst')

That's why we narrowed our data set of over 4 million titles to just 17,546 titles that meet our criteria. This will obviously make the scraping process much faster. Let's complete this list of Title IDs so we don't have to do this cleanup every time:

# Save our data to a "pickle" file, which can be quickly loaded later in the scraping process. values, gherkin)

import scrapyimport pickleclass IMDBSpider(scrapy.Spider): name = 'imdb_spider' # Use the scrape delay to avoid overloading the IMDB servers! custom_settings = { "DOWNLOAD_DELAY": 3, "CONCURRENT_REQUESTS_PER_DOMAIN": 3, "HTTPCACHE_ENABLED": True } # Load the list of title IDs from the pickle file # created in the previous section with open("../my_data.pkl ", 'rb') as a picklefile: links = list(pickle.load(picklefile)) # Create the list of urls to scrape (based on title id) start_urls = [ 'http://www.imdb.com/title / % s /' % l before l in links ] # Methods go here!

For this example, we know that the title of the movie is contained in the

header element and we also know an attribute of the element. We can define our xpath as follows:

So in the parsing code we would have the following line:

title = respons.xpath('//h1[@itemprop="name"]/text()').extract()[0].replace('\xa0','')

def parse(self, answer): # Extraheer de links naar de individualpagina festivalpagina's als 'Box Office' in answer.xpath('//h3[@class="subheading"]/text()').extract(): title_id = respons.url.split('/')[-2] título = respons.xpath('//h1[@itemprop="name"]/text()').extract()[0].replace( ' \xa0','') release = respons.xpath('//div[@class="subtext"]/a/text()').extract()[0].replace('\n',' " ) probeer: rating = answer.xpath('//meta[@itemprop="contentRating"]/@content').extract()[0] behalve: rating = '' probeer: director = answer.xpath('/ / span[@itemprop="σκηνοθέτης"]/a/span[@itemprop="name"]/text()').extract()[0] behalve: director = '' probeer: studio = answer.xpath(' / /span[@itemprop="creador"][@itemtype="http://schema.org/Organization"]/a/span[@itemprop="name"]/text()').extract()[ 0 ] comportarse: estudio = '' dinero = respuesta.xpath('//h3[@class="subheading"]')[0].xpath('volgende broer::div/text()').re( r' \$[0-9,]+') money_labels = answer.xpath('//h3[@class="subheading"]')[0].xpath('volgende-zuster::div/h4/text() ').extract() moneys = [i.replace(',','').replace('$','') for i in money] presupuesto = '' άνοιγμα = '' bruto = ''worldwide_bruto = ' ' probeer: για m, l σε zip(dinero, dinero_etiquetas[:len(dinero)]): if 'presupuesto' en l.inferior(): presupuesto = m elif 'apertura' en l.inferior(): άνοιγμα = m elif 'wereldwijd' in l.lower():worldwide_gross = m elif 'bruto' in l.lower(): bruto = m else: ga verder behalve: pass try: metacritic_score = response.xpath('//div[@ class ="titleReviewBarItem"]/a/div/span/text()').extract()[0] behalve: metacritic_score = '' απόδοση { 'title_id': title_id, 'title': title, 'release': release , 'director': regisseur, 'estudio': estudio, 'presupuesto': presupuesto, 'apertura': άνοιγμα, 'bruto': bruto, 'worldwide_gross':worldwide_gross, 'metacritic_score': metacritic_score, 'mpaa_rating': beoor

To run Scrapy's rotation, the following command is called on the command line

Rastreo scrapy imdb_spider -o 'import_data.json'

This command takes a while to run, but the spider will search all pages for movie titles stored in our pickle file and save the parsed results in json format (in the file called 'import_data.json').

[{"title_id": "tt0315642", "title": "Wazir", "release": "January 8, 2016 (VS)", "director": "Bejoy Nambiar", "studio": "Vinod Chopra Productions ", "budget": "", "opening": "586028", "gross": "586028", "worldwide_gross": "", "metacritic_score": "", "mpaa_rating": ""},{"title_id ": "tt0339736", "title": "The Evil Within", "release": "August 30, 2017 (Indonesia)", "director": "Andrew Getty", "studio": "Supernova LLC", " budget": " 6000000", "opening": "", "gross": "", "worldwide_gross": "", "metacritic_score": "", "mpaa_rating": "UNRATED"},{"title_id": "tt0365907", "title": "A walk among the tombstones", "launch": "September 19, 2014 (VS)", "regisseur": "Scott Frank", "studio": "1984 Private Defense Contractors ", "budget": "28000000", "opening": "12758780", "gross": "26307600", "worldwide_gross": "", "metacritic_score": "57", "mpaa_rating": "R"}, {"title_id": "tt03 " ", "title": "Jurassic World", "release": "June 12, 2015 (VS)", "regisseur": "Colin Trevorrow", "studio": "Universal Pictures ", "budget": "150000000", "opening": "208806270", "gross": "652270625", "worldwide_gross": "1670400637", "metacritic_score": "59", "mpaa_rating": "PG-13 "},{"title_id ": ", "title": "American Pastoral", "release": "October 21, 2016 (VS)", "regisseur": "Ewan McGregor", "studio": "Lakeshore Entertainment ", "budget": " ", " open ": "149038", "gross": "541457", "global gross": "", "meta_rating": "43", "mpaa_rating": "R"}, {"title_id": "tt0403935", "title" ": "Action Jackson", "release": "5 Dec 2014 (India)", "regisseur": "Prabhudheva", "studio": "Baba Films" , "budget": "", "opening": " 171795 ", "gross": "171795", "worldwide_gross": "", "metacritic_score": "", "mpaa_rating": "UNRATED"},{" title_id": "tt0420293", " title": "The Stanford Prison Experiment", "release": "July 17, 2015 (VS)", "regisseur": "Kyle Patrick Alvarez", "studio": "Coup d 'Etat Films", "budget": "", "opening": "37514", "gross": "643557", "worldwide_gross": "", "metacritic_score": "67", "mpaa_rating": "R" },{"title_id": "tt0435651 ", "title": " The Giver", "release": "15 Aug 2014 (VS)", "regisseur": "Phillip Noyce", "studio": "Tonik Productions", "budget": "25000000", "opening": "12305016", "gross": "45089048", "worldwide_gross": "", "metacritic_score": "47", "mpaa_rating": "PG-13 "},...

Fearful! After running this script, there are about 6,000 movie titles left for which box office data is available. It may still be more data than we really need for this project, but I think we can filter it down a bit more before we start scraping off the selenium...

# Load the scrapy results from the IMDB title # Read json fileimport jsonimport panda's as pd # Load scrapy json into my_datawith open('imdb_spider/import_20Jan18_5.json', 'r') as f: my_data = json.load(f)imdb. (my data)

# Filter only useful data (ignore empty values ​​or rare ratings) imdb_mask = ((imdb_info['budget'] != '') & (imdb_info['opening'] != '') & (~imdb_info[' mpaa_rating ' ] .isin(['', 'UNRATED', 'UNRATED', 'TV-14'])))imdb_info = imdb_info[imdb_mask]imdb_info.filter('mpaa_rating NOT ON ["UNRATED", "UNRATED" ", "TV-14]')# Convert columns to usable data typesimdb_info['budget'] = imdb_info['budget'].apply(int)imdb_info['opening'] = imdb_info['opening'].apply( int) imdb_info[ 'launch'] = pd.to_datetime(imdb_info['launch'].apply(lambda x: x.split('(')[0].strip()))# Add new columns to plot(info financial in millions )imdb_info['budget_mil'] = imdb_info['budget']/1000000.imdb_info['opening_mil'] = imdb_info['opening']/1000000.# Rename the Title ID column to 'tconst ' for consistency with other array data [ ' tconst'] = imdb_info['title_id']imdb_info.drop('title_id', inplace=True, as=1)imdb_info.head()

Now we have around 750 movies. This gives us a good dataset to work with (not too big to take the next step with Selenium, but enough data to build our model). Now we are going to merge this DataFrame with the list of movie titles, we need to have a list of unique cast members who were in those movies.

# Combinar con datos IMDB sin procesar usando open('my_data.pkl', 'rb') como picklefile: title = pickle.load(picklefile)titles_all = imdb_info.merge(titles, on='tconst')

Now intitles_allDataFrame, we've linked a list of movie titles with their main cast and crew members (labeledmain cast). What we really want to extract here is a list of unique actors and crew members for all the movies of interest, but we want to associate those actors with each movie for subsequent merges and merges. We can achieve this simply by looking at the title IDs (constant)and related cast lists (main cast), and expand this DataFrame so that there is a single row for each title ID and name ID combination.

col_names = ['tconst', 'principalCast']expanded_data = []for idx, queue en titles_all[col_names].iterrows(): para nombre en queue['principalCast'].split(','): expand_data.append( [queue['tconst'], name.strip()])data_expanded = pd.DataFrame(data_expanded, columnas=['tconst', 'nconst'])

We now have a list of all the actors we need more information on, as well as a breakdown of how these actors are connected to each movie! This simple data framework will prove extremely useful in the next few steps, where we will pull data from IMDB's STARmeter and aggregate that data in a way that will help us build our model.

Through an IMDB Pro account, we can track the popularity of a particular actor on IMDB, using the actor's STARmeter rating. This ranking, given weekly to each cast or crew member featured on IMDB, reflects how often a particular cast member appears in user searches. The IMDB site perfectly sums up the meaning of the STARmeter rating:

def launch_selenium(names_list): # Inicie el navegador web y conéctese a IMDB Pro # Devuelve el objeto del controlador para usar en el proceso de raspado de selenium import webdriver from selenium.webdriver.common.keys import keys from selenium.webdriver.support import added_conditions if Import EC o import time import SENSIBLE as SENS # mv chrome driver from Descargas en aplicaciones chromedriver = "/Applications/chromedriver" os.environ["webdriver.chrome.driver"] = url chromedriver = 'https://pro-labs. imdb .com/name/' + lista_de_nombres[0] + '/' driver = webdriver.Chrome(chromedriver) driver.get(url) loginButton = driver.find_element_by_xpath('//a[@class="log_in"]' ) loginButton .click() time.sleep(.5) loginButton = driver.find_element_by_xpath('//input[@id="auth-lwa-button"]') loginButton.click() time.sleep(.5) nombre_usuario = controlador .find_element_by_id("ap_email") nombre_de_usuario.send_keys(SENS.nombre de usuario) password_form=driver.find_element_by_id('ap_password') password_form.send_keys(SENS.password) password_form.send_keys(Keys.RETURN)

Functionrelease_seleniumit is used to launch my selenium-powered web browser, log into my Amazon account (to access IMDB-Pro content), and navigate to the first web page I'm going to create, based on the cast list and workshop I'm I'm interested. In general, navigating the login menu involves following a similar pattern.

boton = conductor.find_element_by_xpath('

The actual code for this is quite simple and consists of exploring the "rect" tags in the sort chart, clicking each item (to simulate a scroll), and extracting and parsing the "tspan" tag that contains the songs we are interested in. . . The code looks like this:

graph_div = driver.find_element_by_id('ranking_graph')ubicación = graph_div.find_elements_by_tag_name('rect')[1:]name = (driver.find_elements_by_class_name('display-name')[1] .find_element_by_text star_meter_data = []por ejemplo (1, len(localizar)+1): loc = graph_div.find_elements_by_tag_name('rect')[i] (driver.find_element_by_class_name('current_rank') .find_element_by.liks' ()) probeer: loc.click() behalve: time.sleep(0.5) g = graph_div.find_elements_by_tag_name('tspan')[-2:] fechas = g[0].text.split('-') start_date = datetime.datetime.strptime(dates[0].strip (), '%b %d, %Y') end_date = datetime.datetime.strptime(dates[1].strip(), '%b %d, %Y ') star_meter = int(g[1].text .split(':')[-1] .strip() .replace(',','')) star_meter_data.append([i, name_id, name, start_date , einddatum, ster_meter])