Scraping 20.06.08
Project Requirement:
* The successful applicant will be able to complete this project under deadline
* Applicant to provide estimated time to project completion in quote
* This project consists of 3 modules:
1. Relevant Forum Finder
Module to read keywords from a database and update the database of forum urls per keyword.
2. Forum Scraper
Module to scrape all threads and update the database per keyword with the following structure:
- Thread Page Title
- Thread Page Description (to be extracted from the first post in the thread)
- Thread Page Keywords (to be collected by doing a keyword scoring from page content)
- Thread Page Summary (to be extracted from the entire thread page)
- Thread Page URL
- Thread Page Comment Count
3. Distribution Function
Module to provide a platform to query by url and return XML, e.g.
* http://gimme.domain.tld/query/$query-keyword
XML response schema:
<Thread>
<ThreadTitle>Debt Consolidation Loans Home Student Loan Insurance Payday Loans</ThreadTitle>
<ThreadDescription>Consolidation loans and debt consolidation mortgages, just to name a few. Got in a little trouble without the need of any type of home owner or personal loans</ThreadDescription>
<ThreadKeywords>Debt, Consolidation, Loans, Home, Student, Loan, Insurance, Payday, Loans., ForumFix, General, Discussion, Forum</ThreadKeywords>
<ThreadSummary> Debt Consolidation Loan Company We all are currently living in the era of debt consolidation programs . This is entirely due to the ever-increasing demand for these services . There are over thirty million consumers who have below-average credit . Debt management companies are thriving nowadays because of a genuine demand for debt related services as more and more people are now seeking professional help and guidance in terms of debt consolidation and management. Debt consolidation loans are a convenient way to pay off your debts. By doing a little bit of research, you can easily choose a company that will provide the most ethical and fair solution to your debt related problems .</ThreadSummary>
<ThreadURL>http://somecname.somedomain.tld/showthread.php?t=4132326</ThreadURL>
<ThreadCommentCount>7</ThreadCommentCount>
</Thread>
