Web Scraping data from different sites

I am looking for a few ideas on how can I solve a design problem I'm going to be faced with building a web scraper to scrape multiple sites. Writing the scraper(s) is not the problem, matching the data from different sites (which may have small differences) is.

For the sake of being generic assume that I am scraping something like this from two or more different sites:

    public class Data {
        public int id;
        public String firstname;
        public String surname;
        ....
    }

If i scrape this from two different sites, I will encounter the situation where I could have the following:

Site A: id=100, firstname=William, surname=Doe

Site B: id=1974, firstname=Bill, surname=Doe

Essentially, I would like to consider these two sets of data the same (they are the same person but with their name slightly different on each site). I am looking for possible design solutions that can handle this.

The only idea I've come up with is scraping the data from a third location and using it as a reference list. Then when I scrape site A or B I can, over time, build up a list of failures and store them in a list for each scraper so that it can know (if i find id=100 then i know that the firstname will be William etc). I can't help but feel this is a rubbish idea!

If you need any more info, or if you think my description is a bit naff, let me know!

Thanks,

DMcB


Source: http://stackoverflow.com/questions/23970057/web-scraping-data-from-different-sites

Scrape Data Point Using Python

I am looking to scrape a data point using Python off of the url http://www.cavirtex.com/orderbook .

The data point I am looking to scrape is the lowest bid offer, which at the current moment looks like this:

<tr>
 <td><b>Jan. 19, 2014, 2:37 a.m.</b></td>
 <td><b>0.0775/0.1146</b></td>
 <td><b>860.00000</b></td>
 <td><b>66.65 CAD</b></td>
</tr>

The relevant point being the 860.00 . I am looking to build this into a script which can send me an email to alert me of certain price differentials compared to other exchanges.

I'm quite noobie so if in your explanations you could offer your thought process on why you've done certain things it would be very much appreciated.

Thank you in advance!

Edit: This is what I have so far which will return me the name of the title correctly, I'm having trouble grabbing the table data though.

import urllib2, sys
from bs4 import BeautifulSoup

site= "http://cavirtex.com/orderbook"
hdr = {'User-Agent': 'Mozilla/5.0'}
req = urllib2.Request(site,headers=hdr)
page = urllib2.urlopen(req)
soup = BeautifulSoup(page)
print soup.title



Here is the code for scraping the lowest bid from the 'Buying BTC' table:

from selenium import webdriver

fp = webdriver.FirefoxProfile()
browser = webdriver.Firefox(firefox_profile=fp)
browser.get('http://www.cavirtex.com/orderbook')

lowest_bid = float('inf')
elements = browser.find_elements_by_xpath('//div[@id="orderbook_buy"]/table/tbody/tr/td')

for element in elements:
    text = element.get_attribute('innerHTML').strip('<b>|</b>')
    try:
        bid = float(text)
        if lowest_bid > bid:
            lowest_bid = bid
    except:
        pass

browser.quit()
print lowest_bid

In order to install Selenium for Python on your Windows-PC, run from a command line:

pip install selenium (or pip install selenium --upgrade if you already have it).

If you want the 'Selling BTC' table instead, then change "orderbook_buy" to "orderbook_sell".

If you want the 'Last Trades' table instead, then change "orderbook_buy" to "orderbook_trades".

Note:

If you consider performance critical, then you can implement the data-scraping via URL-Connection instead of Selenium, and have your program running much faster. However, your code will probably end up being a lot "messier", due to the tedious XML parsing that you'll be obliged to apply...

Here is the code for sending the previous output in an email from yourself to yourself:

import smtplib,ssl

def SendMail(username,password,contents):
    server = Connect(username)
    try:
        server.login(username,password)
        server.sendmail(username,username,contents)
    except smtplib.SMTPException,error:
        Print(error)
    Disconnect(server)

def Connect(username):
    serverName = username[username.index("@")+1:username.index(".")]
    while True:
        try:
            server = smtplib.SMTP(serverDict[serverName])
        except smtplib.SMTPException,error:
            Print(error)
            continue
        try:
            server.ehlo()
            if server.has_extn("starttls"):
                server.starttls()
                server.ehlo()
        except (smtplib.SMTPException,ssl.SSLError),error:
            Print(error)
            Disconnect(server)
            continue
        break
    return server

def Disconnect(server):
    try:
        server.quit()
    except smtplib.SMTPException,error:
        Print(error)

serverDict = {
    "gmail"  :"smtp.gmail.com",
    "hotmail":"smtp.live.com",
    "yahoo"  :"smtp.mail.yahoo.com"
}

SendMail("your_username@your_provider.com","your_password",str(lowest_bid))

The above code should work if your email provider is either gmail or hotmail or yahoo.

Please note that depending on your firewall configuration, it may ask your permission upon the first time you try it...



Source: http://stackoverflow.com/questions/21217034/scrape-data-point-using-python

Has It Been Done Before? Optimize Your Patent Search Using Patent Scraping Technology

Has it been done before? Optimize your Patent Search using Patent Scraping Technology.

Since the US patent office opened in 1790, inventors across the United States have been submitting all sorts of great products and half-baked ideas to their database. Nowadays, many individuals get ideas for great products only to have the patent office do a patent search and tell them that their ideas have already been patented by someone else! Herin lies a question: How do I perform a patent search to find out if my invention has already been patented before I invest time and money into developing it?

The US patent office patent search database is available to anyone with internet access.

US Patent Search Homepage

Performing a patent search with the patent searching tools on the US Patent office webpage can prove to be a very time consuming process. For example, patent searching the database for "dog" and "food" yields 5745 patent search results. The straight-forward approach to investigating the patent search results for your particular idea is to go through all 5745 results one at a time looking for yours. Get some munchies and settle in, this could take a while! The patent search database sorts results by patent number instead of relevancy. This means that if your idea was recently patented, you will find it near the top but if it wasn't, you could be searching for quite a while. Also, most patent search results have images associated with them. Downloading and displaying these images over the internet can be very time consuming depending on you internet connection and the availability of the patent search database servers.

Because patent searches take such a long time, many companies and organizations are looking ways to improve the process. Some organizations and companies will hire employees for the sole purpose of performing patent searches for them. Others contract out the job to small business that specialize in patent searches. The latest technology for performing patent searches is called patent scraping.

Patent scraping is the process of writing computer automated scripts that analyze a website and copy only the content you are interested in into easily accessible databases or spreadsheets on your computer. Because it is a computerized script performing the patent search, you don't need a separate employee to get the data, you can let it run the patent scraping while you perform other important tasks! Patent scraping technology can also extract text content from images. By saving the images and textual content to your computer, you can then very efficiently search them for content and relevancy; thus saving you lots of time that could be better spent actually inventing something!

To put a real-world face on this, let us consider the pharmaceutical industry. Many different companies are competing for the patent on the next big drug. It has become an indispensible tactic of the industry for one company to perform patent searches for what patents the other companies are applying for, thus learning in which direction the research and development team of the other company is taking them. Using this information, the company can then choose to either pursue that direction heavily, or spin off in a different direction. It would quickly become very costly to maintain a team of researchers dedicated to only performing patent searches all day. Patent scraping technology is the means for figuring out what ideas and technologies are coming about before they make headline news. It is by utilizing patent scraping technology that the large companies stay up to date on the latest trends in technology.

While some companies choose to hire their own programming team to do their patent scraping scripts for them, it is much more cost effective to contract out the job to a qualified team of programmers dedicated to performing such services.

Source:http://ezinearticles.com/?Has-It-Been-Done-Before?-Optimize-Your-Patent-Search-Using-Patent-Scraping-Technology&id=171000

Business Intelligence Data Mining

Data mining can be technically defined as the automated extraction of hidden information from large databases for predictive analysis. In other words, it is the retrieval of useful information from large masses of data, which is also presented in an analyzed form for specific decision-making.

Data mining requires the use of mathematical algorithms and statistical techniques integrated with software tools. The final product is an easy-to-use software package that can be used even by non-mathematicians to effectively analyze the data they have. Data Mining is used in several applications like market research, consumer behavior, direct marketing, bioinformatics, genetics, text analysis, fraud detection, web site personalization, e-commerce, healthcare, customer relationship management, financial services and telecommunications.

Business intelligence data mining is used in market research, industry research, and for competitor analysis. It has applications in major industries like direct marketing, e-commerce, customer relationship management, healthcare, the oil and gas industry, scientific tests, genetics, telecommunications, financial services and utilities. BI uses various technologies like data mining, scorecarding, data warehouses, text mining, decision support systems, executive information systems, management information systems and geographic information systems for analyzing useful information for business decision making.

Business intelligence is a broader arena of decision-making that uses data mining as one of the tools. In fact, the use of data mining in BI makes the data more relevant in application. There are several kinds of data mining: text mining, web mining, social networks data mining, relational databases, pictorial data mining, audio data mining and video data mining, that are all used in business intelligence applications.

Some data mining tools used in BI are: decision trees, information gain, probability, probability density functions, Gaussians, maximum likelihood estimation, Gaussian Baves classification, cross-validation, neural networks, instance-based learning /case-based/ memory-based/non-parametric, regression algorithms, Bayesian networks, Gaussian mixture models, K-means and hierarchical clustering, Markov models and so on.

Source:http://ezinearticles.com/?Business-Intelligence-Data-Mining&id=196648

SEO Copywriting Training - 3 Reasons to Learn This Lucrative Form of Freelance Writing

Here we're going discuss why SEO copywriting is an excellent career choice for anyone who wants to make good money, work from home and get started almost immediately.

Freelance writing has a reputation for being a low-paying career where jobs are hard to come by. It's almost like some type of professional caste system where some seem to make a lot of money effortlessly, while others scrape by on anything they can get (eg, $3 article writing gigs).

It doesn't have to be this way because there's a new type of writing taking over the web. And, if you know how to do it, you can easily make $100-$250+/day. What is it?

SEO Copywriting.

Following are three excellent reasons to train for this lucrative career.

1. Never Ending Need for Content: Anywhere from 30,000 to 100,000 new websites go live every day. This means the web is an increasingly crowded marketplace. Hence, businesses have to find ways to increase their chances of being found online.

Smart business owners, webmasters and bloggers know that SEO content is one of the best ways to go about it. So they invest in this type of content. And, as they constantly need fresh content to stay ahead of the competition, this means freelance writers who know how to provide it stay busy.

2. It's Flexible: Like freelance writing in general, SEO copywriting is a very flexible career. You can work from a coffee shop in New York, a hotel terrace in Jamaica, or a restaurant in South Beach, Miami. As long as there's an internet connection, you can get your work done.

An added bonus in SEO copywriting is that there's very little client interaction. Once they provide you with their keyword phrases, clients rarely need to be in contact with you. They give you keywords and a deadline, and you write the copy and return it via email. Beautiful!

3. Less Competition: There are many freelance writers -- hundreds of thousands, perhaps even a few million. But, very few of them know how to write SEO content. This means there's less competition for these types of writing assignments.

Just how important is an SEO copywriter to webmasters, bloggers, internet marketers -- and/or anyone who wants to be found online?

ln the SearcheEngineWatch article, The SEO Copywriter: Wordsmithing the WebWillam Flaiz underscores the importance of the SEO copywriter. He writes:

The role of the agency SEO copywriter is unique because. . . . Suddenly, the writer must optimize press releases, craft articles for social news, monitor social network communications, and develop wiki content. . . . a strong SEO copywriter is adaptable and capable of internalizing a lot of new information extremely quickly. These unique individuals have fast become the backbone of many SEO engagements. If content is king, then they [SEO copywriters] are the kingmakers.

If you want a high-paying , work-from-anywhere you please career, get SEO copywriting training -- and be well on your way to running your own successful freelance writing business.

Source: http://ezinearticles.com/?SEO-Copywriting-Training---3-Reasons-to-Learn-This-Lucrative-Form-of-Freelance-Writing&id=2943036

SEO Epiphany of an Offline Marketer Turned Online Marketer

SEO can be maddening for offline marketers such as myself transitioning to the online marketing world. In making this transition I just had an epiphany sparked by a discussion I just had with one of my Skype discussion groups. In offline marketing we are taught to use emotionally charged words depending on our targeted market. Words that evoke emotions such as greed, fear, love, hate, pride etc. are utilized throughout our marketing and is probably equivalent to keywords in online marketing.

When I sold Real Estate for buyers we focused a lot on pride, in my tax business fear was a dominating emotion especially fear of the IRS and fear of audits. In my tax business I specialized in Home Based Business owners and my two biggest client groups were Daycare Providers and Network Marketers. The most effective emotion in the Network Marketing industry was greed.

In offline marketing we send our advertising out through mediums that get us the best results. Newspapers and periodicals are sent to people's homes, flyers are put into people's hands or on their door our car, radio is broadcast to people listening to their radios. We learn how to get a response through our ads with these emotionally charged words.

Then we are taught how to write copy that not only uses these emotionally charged words but we are taught how to use tie downs like wouldn't it, couldn't it, shouldn't it. We are taught how to develop closing sequences that get lots of little yeses leading to the big yes.

Then we come to the online marketing arena and all the rules change. At first the changes are not obvious. We hear familiar terms like call to action and compelling headlines. But unlike in the offline marketing arena it appears that to play in this game you have to contend with this huge monster called Google. And this creates a huge paradigm shift in how one has to develop their marketing tactics. Offline we send our stuff out, online you have to be found. And you have to be found through Google. Bing does not really matter; Yahoo does not really matter only the Google search engine at this time really matters.

Sure there are advertising sites, Google AdWords, Craigslist and Facebook Classified ads and who knows what else is bound to pop up but none of these are true direct response marketing and function more like billboards and posters in high traffic areas. The closes thing to marketing material being sent directly to a person is email marketing which due to spam and individuals emails being overwhelmed with tons of stuff it is becoming increasingly ineffective unless you have a dedicated list looking forward to stuff you are sending them.

It seems that in order to market successfully on line you have to find out two things - what words or phrases that people are using to search for stuff on Google and what words and phrases that Google does not like. Online copywriters don't so much worry about closing sequences as they do Search Engine Optimization often called SEO. That means writing copy in such a way that Google spiders will find you and elevate you to the top of the heap on their clients searched words or phrases.

Besides such little things as meta tags, title tags, h1 tags and making sure your keywords and phrases are at the top of your copy, SEO is about two key elements. One is keywords, you know those words or phrases that people plug into Google to search for stuff and secondly Google perceived relevancy of your copy. Two websites being equal as far as keywords are concerned Google's perception of relevancy will determine who gets the top spots.

The precise keywords and phrases have to be strategically placed in your copy. But you have to be careful because certain words and phrases it seems Google has a hatred for and will band you. Keywords such as riches, wealth, success and money are among these words Google hates and could possibly even penalize you for using by forbidding your site from ever showing in their search results (I have a lot to say about such power being in the hands of one company but not today)

The density or the amount of times your selected keywords or phrases are placed in your copy becomes critical, too often and you will be penalized, not enough and you won't be recognized. Best rule of thumb is to use them were it makes sense in your copy. Trying to figure out percentages is simply not working with today's advanced algorithms. Best to focus on content with the understanding that if your key words or phrases are not in your copy then when people search for it you will not be found so where it makes sense you must use them.

Next is relevancy despite all the stuff we hear about backlinks there is far more to being determined as relevant by Google search engines than backlinks. Simple things such as being listed in the yellow pages can make your site more relevant than another site. Whereas backlinks are still very important you can have link spam that could make you irrelevant with Google. This topic is just too extensive a topic to cover here, besides this is about an epiphany concerning my transition from offline marketer to full time online marketing. But I will add this tidbit that I have learned about being perceived as relevant with Google. Avoid duplicate content. websites with identical pages, scraped content, heavily distributed articles and boiler plate pages/sites can hurt your relevancy perception with Google.

Even when you are the original author putting the exact same article into e-zines and blogs can hurt your relevancy if the same articles are posted on the websites you want to come up in the Google search engines. On the sites you want to rank in Google keep all content as original and unique as possible.

Source: http://ezinearticles.com/?SEO-Epiphany-of-an-Offline-Marketer-Turned-Online-Marketer&id=6713183

Kerry Finch Writing Announces Streamlining of Content Writing Services

Content writing business, Kerry Finch Writing, has announced the restructuring of the services it provides to businesses around the world. In revealing the changes, owner, Kerry Finch, gave some insights into the reasons behind the decision.

“The digital business world is dynamic in the true sense of the word: it is constantly changing. Rather than simply reacting to those changes, it makes better business sense to anticipate them, and to be nimble in the implementation of newer and more effective strategies,” said Ms Finch. “Because I am active in several vibrant online business communities, and lead a tightly knit, responsive, team, we are in a great position identify new opportunities and to apply them quickly and seamlessly to the content we are writing.”

“Very often my clients don’t even recognize the often subtle changes in the written content we deliver to them, for instance the increased use of niche-relevant semantics, the more engaging ‘calls to action’, the up-to-the-minute topic selections, and, where appropriate, moving towards more geo-targeted content,” she continued. "Geo-targeting is becoming very important for connecting with people using mobile devices for search queries."

“All of the modifications are implemented with a focus on current best-practices.”

In redefining the types of blogs offered by Kerry Finch Writing, Ms Finch has her sights on keeping her business, and those of her clients, both competitive and relevant.

Blog Posts

The changes include removing the term ‘articles’ from her product offering, and replacing these with ‘blog posts’.

“Seven years ago, when I established Kerry Finch Writing, a key service I provided was article writing. These were often published on article directories with the objective of acquiring valuable links back to websites, but things have changed dramatically since then,” Ms Finch explained.

“These days I help each of my business clients, regardless of their size, to understand the need to grow their own websites with original, relevant content. I help them to understand that their website is a business asset with a dollar value - but unlike other assets, if it is well-maintained, it will appreciate over time - not depreciate.”

“Once a website has been built and populated with well written content, the best way for it to grow is by having a News or Blog section, which we can feed with new posts on a regular basis. Those clients who have a recurring monthly blog post order with us, are the ones who are reaping the rewards - their websites are being found by the search engines, and they are enjoying better engagement with their ideal customers.”

Kerry Finch Writing now offers three types of blog posts: Evergreen (general niche-relevant posts), News (relating a current news story to the clients’ businesses), and Authority (which are longer, more deeply researched industry-focused posts).

“News-focused blog posts are now one of the most effective and types of content that we offer. They demonstrate to readers that the blog is current, and that the business owner is aware of what’s happening in the real world. These posts also have the added benefit of riding off the popularity of trending news stories,” said Ms Finch. “The practice of not dating blog posts is old-school, people need to see that what they are reading is accurate right now, not 2, 3 or 4 years ago - and this is especially important for my many law firm clients.”

Removal of Services

Some of the services that Kerry Finch Writing has provided in the past are no longer being offered, but the decision to remove them are also seen as a positive step.

“I am not in the business of offering some writing services just because we can. If a particular type of content is not delivering results, then I’d rather not include it in our suite of services,” said Ms Finch.

“The more streamlined content options we are now providing have been carefully considered, and as new opportunities arise, I will no doubt be changing them again. For instance I see more downloadable White Papers in the near future. Being adaptable is vital to succeeding in the online world.”

Source: http://www.virtual-strategy.com/2014/06/26/kerry-finch-writing-announces-streamlining-content-writing-services