How To Set Up A Bot For Ps5
Introduction
I've never had a gaming console my entire life (PSP doesn't count). It looks similar it's the best time to alter it thanks to the recent release of PS5 and Xbox Series 10. My eyes are primarly focused on the newest PlayStation due to its exclusive titles, such equally: Spiderman, The Terminal of Us, Uncharted, etc.
However, I didn't preorder it, since it turned out to be a gamble. One shop delivered some preorders, but another said they volition take them only in January. I don't want to take a consolless Christmas, and so my plan was to grab information technology during the starting time twenty-four hours of sale. Unfortunately, I was non quick enough :(
Some online shops offer signing up for a newsletter that would hopefully notify me if there is restock. However, giving my mail to them is equal to receiving huge amount of spam, and the unsubscription doesn't necessarily mean they will delete my email. In the nearly future the sale will be entirely online.
Another mode to get the console is through people who already bought them. Simply the prices... They are 2x more expensive (in the store they price 2200).
I was actually pissed! There are then many people, who bought the panel only to resell them correct after for the higher cost, while there are and then many, who want to just enjoy playing the games. Capitalism, right?
Goal
Fortunately, when I'm pissed I'g also very motivated. It would also be absurd to combine information technology with a valuable skill called programming to achieve the goal:
To buy a PS5 before Christmas
In club to help me with that I wrote a bot that scraps PS5 product pages of several polish online shops. After detecting that their availability changed information technology notifies me, and so I can manually get to the shop and buy it.
It's only a change detection bot and not some auto buyer.
Here is a sneak peek of how information technology looks:
Enquiry
The approach I took is basically to fetch the page every 5 minutes and check if at that place are strings indicating something changed. For instance in one instance I check for a literal text 'The production is temporarily available' while in another one I cheque for a characteristic class name.
I've targeted 7 online polish shops. Later some research (clicking the site and inspecting network requests) I noticed some differences I need to take into consideration before staring to code.
-
HTML vs JSON - Some shops use a SSR (Server Side Rendering), so all the content is directly embedded into HTML file. Nevertheless, some fetch the data using AJAX in JSON format.
-
Inconsistent product pages - Some shops don't result have a PS5 product page all the same, so they utilize a fancy landing page, some have a production page, and ane shop doesn't have either, so its only indication is that the search list is empty.
In
Avansnosotros tin can just check if there is no PS5 on the listing.
In
MediaMarktwe can only see a landing folio.
Site definitions
I've written the bot in Node.js using Typescript. The structure of the project looks like this:
Every shop has a dedicated course, which allows to arrange some quirks per shop. Each shop definition looks like this:
// SITE WITH SSR // Detect information technology extends from HTML consign course KomputronikDef extends HtmlSiteDef { protected getConfig (): SiteConfig { return { name : ' Komputronik ' , url : ' https://www.komputronik.pl/product/701046/sony-playstation-five.html ' , }; } // Notice it receives a Document as a parameter protected hasUnexpectedChanges ( certificate : Document ): boolean { const phrase = ' Produkt tymczasowo niedostępny. ' ; const xPathResult = document . evaluate ( `//*[normalize-infinite() = ' ${ phrase } ']` , document , null , ORDERED_NODE_SNAPSHOT_TYPE , null ); return xPathResult . snapshotLength === 0 ; } }
Each site definition has 2 methods.
-
getConfig()- for a static data -
hasUnexpectedChanges(...)- cadre of the functionality. Here nosotros check for a specific values that would indicate that the production is still not available. Detect it receives aDocumentequally a parameter, which is a parsed DOM tree, just like in a browser, and then nosotros can use some CSS selectors, or like in this instance, XPATH to find a specific string.
In that location is too JSON blazon site definition that looks virtually exactly the same, but instead of receiving a Document equally a parameter it gets a JSON object.
// SITE WITH AJAX Request // Notice it extends from JSON export class NeonetDef extends JsonSiteDef < NeonetResponse > { protected getConfig (): SiteConfig { return { name : ' Neonet ' , url : ' https://www.neonet.pl/graphql?query=query%20landingPageResolver($id:%20Int!)%20%7B%20landingPage:%20landingPageResolver(id:%20$id)%20%7B%20name%20custom_css%20teaser_alt%20teaser_file%20teaser_file_mobile%20show_teaser%20date_from%20clock_type%20modules%twenty%7B%20id%20position%20type%20parameters%twenty%7D%20is_outdated%20%7D%0A%7D%0A&variables=%7B%22id%22:1451%7D&v=2.54.0 ' , }; } // Notice it receives an object specified // in the base class JsonSiteDef<NeonetResponse> protected hasUnexpectedChanges ( json : NeonetResponse ): boolean { return ! this . hasProperTitle ( json ) || ! this . hasThankYouModule ( json ); } individual hasProperTitle ( json : NeonetResponse ): boolean { render json . data . landingPage . proper noun === ' Premiera Konsoli Playstation 5 ' ; } private hasThankYouModule ( json : NeonetResponse ): boolean { const module = json . data . landingPage . modules [ 4 ]; if ( ! module ) { return false ; } /** * Cannot bank check all the message, considering from the backend we become them encoded */ const lastPartOfMessage = ' west celu uzyskania dalszych aktualizacji. ' ; return module . id === 7201 && module . parameters . includes ( lastPartOfMessage ); } }
Custom framework
If you noticed there are 2 base classes HtmlSiteDef and JsonSiteDef that both fetch the site and make either a DOM tree of a JSON object. Below is an instance of HtmlSiteDef.
// Find it too extends from SiteDef export abstract grade HtmlSiteDef extends SiteDef { protected async _internalTriggerChanges (): Hope < void > { // we fetch a page const body = look this . getBodyFor ( this . config . url , this . config . cookie , ' html ' ); // we create a DOM tree const dom = new JSDOM ( torso ); // we invoke an abstract method implemented by a child class const somethingChanged = this . hasUnexpectedChanges ( dom . window . document ); if ( ! somethingChanged ) { this . logger . info ( `Nix inverse...` ); } else { this . logger . warn ( `-----------------------------------` ); this . logger . warn ( `SOMETHING CHANGED!!!` ); this . logger . warn ( `-----------------------------------` ); // we as well send an e-mail this . sendSuccessMail (); } } // hither we define a method to be implemented per site definition protected abstract hasUnexpectedChanges ( document : Document ): boolean ; }
In that location is also a base grade for them all called SiteDef. It's basically responsible for fetching a page and sending a success e-mail, or in example of some exception, such equally blocking ip, invalid response stats, etc., sending an fault email.
export abstract class SiteDef { // the config from the child class protected config = this . getConfig (); protected logger = getLogger ( this . config . proper name ); // more on sending a mail later protected mailSender = new MailSender (); // flags for sending an email, // nosotros desire to send email only once, so that information technology'south non treated as spam private alreadySentMail = simulated ; individual alreadySentErrorMail = false ; // classes for children to implement protected abstract getConfig (): SiteConfig ; protected abstract _internalTriggerChanges (): Promise < void > ; // main method invoked every 5 minutes async triggerChanges (): Promise < void > { endeavour { await this . _internalTriggerChanges (); this . alreadySentErrorMail = faux ; } take hold of ( e ) { this . logger . error ( e ); if ( ! this . alreadySentErrorMail ) { this . alreadySentErrorMail = true ; this . mailSender . sendError ( this . config . name , e ); } } } protected async getBodyFor ( url : string , cookie : string , type : ' json ' | ' html ' ): Promise < string > { // we demand to spoof the headers, then the request looks legitimate const response = look fetch ( url , { headers : { ' User-Agent ' : ' Mozilla/5.0 (Windows NT x.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0 ' , Take : type === ' html ' ? ' text/html ' : ' application/json ' , ' Have-Language ' : ' en-GB,en;q=0.5 ' , Referer : ' https://www.google.com/ ' , Pragma : ' no-cache ' , ' Cache-Command ' : ' no-enshroud ' , ' Have-Encoding ' : ' gzip, deflate, br ' , Cookie : cookie ?? null , }, }); return expect response . text (); } protected sendSuccessMail (): void { if ( ! this . alreadySentMail ) { this . alreadySentMail = truthful ; this . mailSender . send ( this . config . proper name ); } } }
Main loop
Inside index.ts we simply loop the sites lists every 5 minutes.
// 5 minutes const TIMEOUT = 5 * 60 * 1000 ; // list of all the supported sites const sites : SiteDef [] = [ new MediaMarktDef (), new MediaExpertDef (), new NeonetDef (), new EuroDef (), new EmpikDef (), new AvansDef (), new KomputronikDef (), ]; function sleep ( timer : number ): Promise < void > { return new Promise < void > (( resolve ) => setTimeout (() => resolve (), timer )); } // the master infinite loop async role principal () { while ( true ) { for ( const site of sites ) { wait site . triggerChanges (); } panel . log ( ' ------------- SLEEPING ------------- ' ); await sleep ( TIMEOUT ); } } chief ();
Sending an email
Get-go I thought about writing a mobile app that would send me a custom notification, but the same functionality can be achieved merely by sending an electronic mail to my gmail account, which in turn would display a notification on my phone. Cool
For this purpose I used sendgrid mainly because it has a free tier with 100 mails per day, which is 100x more than I need.
Integration was super like shooting fish in a barrel. I took me less than 15 minutes to successfully send the first email.
one. Custom DNS entries
Sendgrid requires a custom domain to be verified by adding some DNS entries. Luckily I accept mine in Cloudflare, so it was a piece of block.
Here is what I had was presented by Sendgrid
Here is where I put the entries on Cloudflare
2. Downloading a Node library
They have a dedicated library, which can be installed with:
npm install --salvage @sendgrid/mail
So on top of it I created a MailSender wrapper class that you might accept noticed in SiteDef grade.
// nosotros set api key created in the sendgrid app sgMail . setApiKey ( process . env . SENDGRID_API_KEY ); export grade MailSender { send ( siteName : string ): void { const mailData : MailDataRequired = { to : procedure . env . TARGET_MAIL , from : process . env . SENDGRID_MAIL , subject : `[ps5-bot] ${ siteName } has changed` , text : ` ${ siteName } has changed` , }; sgMail . ship ( mailData ) . then (() => { logger . info ( ' Mail sent ' ); }) . catch (( error ) => { logger . warn ( mistake ); }); } sendError ( siteName : cord , error : Error ): void { const mailData : MailDataRequired = { to : process . env . TARGET_MAIL , from : process . env . SENDGRID_MAIL , subject : `[ps5-bot] ERROR in ${ siteName } ` , text : ` ${ error . stack } ` , }; sgMail . send ( mailData ) . and so (() => { logger . info ( ' Mail sent ' ); }) . catch (( mistake ) => { logger . warn ( error ); }); } }
It is very simple, it has just two methods, 1 for sending success mail and the other for sending an error. The error message also sends a stack trace of the exception, so that I know which role of code bankrupt. Below is the mistake mail screen.
Y'all can also notice that the bot uses sensitive information, such as: SENDGRID_API_KEY, SENDGRID_MAIL, TARGET_MAIL using surroundings variables. Goose egg is hardcoded.
Deployment
I was thinking almost setting a pipeline, that would build a Docker image, put information technology on DockerHub and then deploy it to Kubernetes cluster using Terraform on my RaspberryPi, however, it would be an overkill. I hope this bot would do its task during the next couple of weeks and be forgotten, so the pipeline doesn't need to be fancy.
This is why I decided to manually SSH into my RaspberryPI, pull the repository and and so run the Docker image. All past paw.
First I created a Dockerfile
FROM node:14.15-alpine as builder WORKDIR /usr/app/ps5-bot Copy ./package.json ./bundle-lock.json ./ RUN npm set progress = false RUN npm ci COPY . . RUN npm run build # ----------- FROM node:xiv.xv-alpine WORKDIR /usr/app/ps5-bot Re-create --from=builder /usr/app/ps5-bot/build build Re-create --from=builder /usr/app/ps5-bot/node_modules node_modules ENTRYPOINT ["node", "./build/main/index.js"]
Then a docker-compose.yml which would allow me to quickly make it running.
version : ' 3' services : ps5-bot : build : context : . restart : ever env_file : - .env
To run it I used a Docker Compose CLI:
docker-compose up -d
Here is the final result:
The repository:
Humberd / ps5-bot
Bot for crawling popular smoothen shops checking for PS5 avalability
Conclusion
Creation of this bot took me 7 hours:
- 5 hours of research and implementation
- 1 60 minutes of configuration and integration with Sendgrid
- one hour of configuring a deployment
I am pretty happy of what I achieved. The bot crawls vii pages every 5 minutes looking for changes and when information technology happens it emails me. It is currently deployed on my RaspberryPi running within a Docker container.
Now I need to patiently wait for an email to come up :)
Brand certain to follow me to accept an update on the result of this project
See you lot once again.
Source: https://dev.to/humberd/how-i-wrote-a-ps5-hunter-bot-in-7-hours-6j4
0 Response to "How To Set Up A Bot For Ps5"
Post a Comment