Go Back   English Forum Switzerland > Help & tips > Other/general  
Reply
 
Thread Tools Display Modes
  #1  
Old 07.05.2021, 17:55
Junior Member
 
Join Date: Oct 2017
Location: Zurich
Posts: 86
Groaned at 2 Times in 2 Posts
Thanked 40 Times in 21 Posts
Rozelina has no particular reputation at present
Any web developer here to give some advice?

Hi everyone.

Hope you are safe and well.

I am a complete noob when it comes to how websites work and what is happening behind the scenes, so I would appreciate if someone can shed some light please. I am working on building a website on my free time and I will need to pay someone at some point to do the IT/engineering stuff but before I get there I am trying to understand simply how things work.

An example: a property valuation website where someone inputs the data of a property ( size, bedrooms, balkony, garden, year of construction, etc..) and the website provides an objective price of that property. This is where my questions start:

1. Does the website already have a database of property prices so it takes the inputs of the user, compares it with the database and provides the average price of similar properties? If this is the case, I assume the database needs to be updated frequently in order to reflect current market prices, right?

2. Or, does the website somehow search live and dynamically the internet for properties being sold (e.g homegate, immoscout24) and provide with an average price based on what similar properties are on the popular property sales website? If this is the case, how is this done? Is this a code a web developer writes which then is scrapping data from other websites? Would that "code" need some kind of maintainance or is this something a web developer writes once and it can be used for years.

I apologize if my questions sound silly but I don't know much about websites. I am a great user of them though

Thank you in advance.
Reply With Quote
  #2  
Old 07.05.2021, 18:01
Member
 
Join Date: Apr 2008
Location: wallisellen
Posts: 196
Groaned at 1 Time in 1 Post
Thanked 252 Times in 91 Posts
irish_marmot has earned the respect of manyirish_marmot has earned the respect of manyirish_marmot has earned the respect of many
Re: Any web developer here to give some advice?

Quote:
View Post
Hi everyone.

Hope you are safe and well.

I am a complete noob when it comes to how websites work and what is happening behind the scenes, so I would appreciate if someone can shed some light please. I am working on building a website on my free time and I will need to pay someone at some point to do the IT/engineering stuff but before I get there I am trying to understand simply how things work.

An example: a property valuation website where someone inputs the data of a property ( size, bedrooms, balkony, garden, year of construction, etc..) and the website provides an objective price of that property. This is where my questions start:

1. Does the website already have a database of property prices so it takes the inputs of the user, compares it with the database and provides the average price of similar properties? If this is the case, I assume the database needs to be updated frequently in order to reflect current market prices, right?

2. Or, does the website somehow search live and dynamically the internet for properties being sold (e.g homegate, immoscout24) and provide with an average price based on what similar properties are on the popular property sales website? If this is the case, how is this done? Is this a code a web developer writes which then is scrapping data from other websites? Would that "code" need some kind of maintainance or is this something a web developer writes once and it can be used for years.

I apologize if my questions sound silly but I don't know much about websites. I am a great user of them though

Thank you in advance.



Option 2. The data you need would be stored somewhere and made available to you. Your application then would query that source (via APIs) to get the latest data. The problem might (probably) will be that you have to pay for that data source. Comparis or search.ch might provide this information via APIs to you.



Just a bit more info for you if you are totally starting out.


You can build a web application quite quickly on Google cloud, Azure or AWS. All of these would start with free accounts for you. My preference is AWS but any would do. Here is an example getting started point but there are loads out there....
https://aws.amazon.com/getting-start...eway-dynamodb/ You have some application code to write (implementing those APIs I mentioned) but not much. The costly part will be getting the data. Thats why data is so valuable
Reply With Quote
The following 2 users would like to thank irish_marmot for this useful post:
  #3  
Old 07.05.2021, 18:45
Member
 
Join Date: Apr 2008
Location: wallisellen
Posts: 196
Groaned at 1 Time in 1 Post
Thanked 252 Times in 91 Posts
irish_marmot has earned the respect of manyirish_marmot has earned the respect of manyirish_marmot has earned the respect of many
Re: Any web developer here to give some advice?

By the way, there are great online courses at reasonable prices that will accelerate your learning. I really like udemy so you could consider this (or look around they may have one that is a better fit):


https://www.udemy.com/course/the-web...oper-bootcamp/


This guy compares some of the options



https://medium.com/javarevisited/10-...s-9942af352ed5


good luck!
Reply With Quote
This user would like to thank irish_marmot for this useful post:
  #4  
Old 07.05.2021, 18:49
Junior Member
 
Join Date: Oct 2017
Location: Zurich
Posts: 86
Groaned at 2 Times in 2 Posts
Thanked 40 Times in 21 Posts
Rozelina has no particular reputation at present
Re: Any web developer here to give some advice?

Quote:
View Post
Option 2. The data you need would be stored somewhere and made available to you. Your application then would query that source (via APIs) to get the latest data. The problem might (probably) will be that you have to pay for that data source. Comparis or search.ch might provide this information via APIs to you.



Just a bit more info for you if you are totally starting out.


You can build a web application quite quickly on Google cloud, Azure or AWS. All of these would start with free accounts for you. My preference is AWS but any would do. Here is an example getting started point but there are loads out there....
https://aws.amazon.com/getting-start...eway-dynamodb/ You have some application code to write (implementing those APIs I mentioned) but not much. The costly part will be getting the data. Thats why data is so valuable

Thank you so much for your answer. Regarding getting the data, how do I approach it? I mean is there a chance the application you mentioned cannot pull data from another website? Can it be blocked for example ? In that case would I need to come in touch with the owner of this website to ask for access via API?
Reply With Quote
  #5  
Old 07.05.2021, 18:58
Member
 
Join Date: Apr 2008
Location: wallisellen
Posts: 196
Groaned at 1 Time in 1 Post
Thanked 252 Times in 91 Posts
irish_marmot has earned the respect of manyirish_marmot has earned the respect of manyirish_marmot has earned the respect of many
Re: Any web developer here to give some advice?

Quote:
View Post
Thank you so much for your answer. Regarding getting the data, how do I approach it? I mean is there a chance the application you mentioned cannot pull data from another website? Can it be blocked for example ? In that case would I need to come in touch with the owner of this website to ask for access via API?

You google it - not being smart - lots of web developement you will spend googling or on stackoverflow which is a forum for IT related questions (like english forum just instead of bickering expats its bickering IT people).


Contact comparis.ch, search.ch, homegate etc. and ask if they offer APIs which list [describe the dataset you need], they will then let you know. Different places have different terms - like we dont sell access to the data for commercial use but will give you APIs to test data sets for education etc. If its real data you are talking about (ie not dummy test data) then remember you got topics like GDPR, data residency etc. to consider, ie they may not give you API access due to data privacy issues to address their regulatory processes.








Reply With Quote
This user would like to thank irish_marmot for this useful post:
  #6  
Old 07.05.2021, 19:04
amogles's Avatar
Forum Legend
 
Join Date: Mar 2008
Location: Zurich
Posts: 12,345
Groaned at 338 Times in 275 Posts
Thanked 26,173 Times in 10,978 Posts
amogles has a reputation beyond reputeamogles has a reputation beyond reputeamogles has a reputation beyond reputeamogles has a reputation beyond reputeamogles has a reputation beyond reputeamogles has a reputation beyond repute
Re: Any web developer here to give some advice?

It's not popular and you won't make yourself friends, but if you want to build a database quickly, screenscraping sites like Homegate may be the way to go.

From my experience with the real estate market, most real estate agents like to guard their little kingdoms and would never share any information with you for free if you're not at least a potential customer. But maybe you can get lucky?

Maybe banks also keep records on property sales and prices and may be more amenable to providing some anonymized version of that. But it would still be a long shot.
Reply With Quote
This user would like to thank amogles for this useful post:
  #7  
Old 07.05.2021, 19:21
NotAllThere's Avatar
Forum Legend
 
Join Date: Oct 2008
Location: Baselland
Posts: 13,612
Groaned at 236 Times in 207 Posts
Thanked 19,972 Times in 8,115 Posts
NotAllThere has a reputation beyond reputeNotAllThere has a reputation beyond reputeNotAllThere has a reputation beyond reputeNotAllThere has a reputation beyond reputeNotAllThere has a reputation beyond reputeNotAllThere has a reputation beyond repute
Re: Any web developer here to give some advice?

Quote:
View Post
You google it - not being smart - lots of web developement you will spend googling or on stackoverflow which is a forum for IT related questions
This is true. I'm an expert professional developer - 30 years+ experience - and I still google for stuff. What's the point of holding the knowledge in my head when it's online?
Reply With Quote
  #8  
Old 07.05.2021, 19:23
blacky's Avatar
Member
 
Join Date: Mar 2021
Location: Bezirk Affoltern, ZH
Posts: 182
Groaned at 5 Times in 3 Posts
Thanked 240 Times in 104 Posts
blacky has an excellent reputationblacky has an excellent reputationblacky has an excellent reputationblacky has an excellent reputation
Re: Any web developer here to give some advice?

Just to address the second point. You asked for case when some code does the dynamic scraping, if it's 'set and forget'. No.
Since all those target sources can and will change sometime in the future. More controllable if you use official APIs eg you'll get a email as a paying customer that they're changing output form, and zero control if you're just doing web scraping - since anyone can and will change their website, variable names and whatever anytime.

Another point, value of the property is generally what market is willing to pay for it.
We can argue that this land x and this building y should go for no less than z.
But I can assure you, there are properties which you buy for the land only, and those which you wouldn't buy for yourself no matter how cheap they are, because of the layout or solutions that were used during the building (eg poor ventilation). So number of rooms and similar few parameters just can't be enough to give you anything close to 'guessing the market value'.

There is a reason why there are those bank/insurance estimators who go into the flat and look around everything to give the information to the bank what's minimal number they should be able to get if you can't pay that mortgage anymore.
We cannot put everything into variables, especially things that depend on humans - and most of the people buy house they like among all those with the same number of rooms.
Reply With Quote
This user would like to thank blacky for this useful post:
  #9  
Old 07.05.2021, 19:23
Forum Veteran
 
Join Date: Nov 2007
Location: Vaud
Posts: 1,013
Groaned at 126 Times in 62 Posts
Thanked 515 Times in 275 Posts
markalex has an excellent reputationmarkalex has an excellent reputationmarkalex has an excellent reputationmarkalex has an excellent reputation
Re: Any web developer here to give some advice?

The saying "garbage in, garbage out" comes to mind. You need to be sure the data source is good, else your results will be bad.

And if your results are bad nobody is going to be going to your website, customers are very unforgiving.
Reply With Quote
The following 2 users would like to thank markalex for this useful post:
  #10  
Old 08.05.2021, 01:23
Jim2007's Avatar
Forum Legend
 
Join Date: Jun 2006
Location: Kt. Bern
Posts: 4,508
Groaned at 217 Times in 171 Posts
Thanked 7,421 Times in 3,283 Posts
Jim2007 has a reputation beyond reputeJim2007 has a reputation beyond reputeJim2007 has a reputation beyond reputeJim2007 has a reputation beyond reputeJim2007 has a reputation beyond reputeJim2007 has a reputation beyond repute
Re: Any web developer here to give some advice?

Quote:
View Post
Thank you so much for your answer. Regarding getting the data, how do I approach it? I mean is there a chance the application you mentioned cannot pull data from another website? Can it be blocked for example ? In that case would I need to come in touch with the owner of this website to ask for access via API?
You really are not asking questions about how to implement your business concept, you are expecting others to actually give you the concept! If you donít have a valuation concept, know what data you need and how to source it, then what do you bring to the table? An ability to copy other peopleís stuff...

People with valuable data are well aware that others will attempt to screen scrap it, so they take precautions to make it very difficult to do it in volume or frequently. You can also expect that these people will not let you have their data for free and they will take legal action if necessary to protect their interests.
__________________
"There is no passion to be found playing small - in settling for a life that is less than the one you are capable of living." - Nelson Mandela
Reply With Quote
The following 2 users would like to thank Jim2007 for this useful post:
  #11  
Old 09.05.2021, 00:34
Junior Member
 
Join Date: Oct 2017
Location: Zurich
Posts: 86
Groaned at 2 Times in 2 Posts
Thanked 40 Times in 21 Posts
Rozelina has no particular reputation at present
Re: Any web developer here to give some advice?

Quote:
View Post
You really are not asking questions about how to implement your business concept, you are expecting others to actually give you the concept! If you donít have a valuation concept, know what data you need and how to source it, then what do you bring to the table? An ability to copy other peopleís stuff...

People with valuable data are well aware that others will attempt to screen scrap it, so they take precautions to make it very difficult to do it in volume or frequently. You can also expect that these people will not let you have their data for free and they will take legal action if necessary to protect their interests.
Hi Jim2007

Thank you very much for taking the time to answer. I understand and I agree with your points mentioned above. The property valuation case is just an example not the concept I am working on. I also know where to get the data. What I donít know is how to get data from website A and move it to website B. I am going to pay someone to do it for me but I am trying to find out what my options are and how feasible they are.

Now, if you are familiar with web developing you must have understood from my post that I donít know much about how websites work. If you are not familiar with web developing, why even bother? Regardless, thanks a lot for your inputs.
Reply With Quote
  #12  
Old 09.05.2021, 04:56
Member
 
Join Date: Feb 2020
Location: Zurich
Posts: 156
Groaned at 20 Times in 9 Posts
Thanked 143 Times in 79 Posts
spark is considered knowledgeablespark is considered knowledgeablespark is considered knowledgeable
Re: Any web developer here to give some advice?

You either talk to site A and compel them to give you API access (if you want to query them live as users come to your site) or the data/feed directly - then you can process it on your servers and serve it from there yourself.

Or you scrape the shit out of them semi-legally without asking and then use the scraped data to fill up your site's databases/models and also host everything yourself.
Reply With Quote
  #13  
Old 09.05.2021, 06:58
Jim2007's Avatar
Forum Legend
 
Join Date: Jun 2006
Location: Kt. Bern
Posts: 4,508
Groaned at 217 Times in 171 Posts
Thanked 7,421 Times in 3,283 Posts
Jim2007 has a reputation beyond reputeJim2007 has a reputation beyond reputeJim2007 has a reputation beyond reputeJim2007 has a reputation beyond reputeJim2007 has a reputation beyond reputeJim2007 has a reputation beyond repute
Re: Any web developer here to give some advice?

Quote:
View Post
What I don’t know is how to get data from website A and move it to website B.
A novel idea perhaps but start by asking the owners if they are willing to let you have it and how they will let you access it.

Quote:
Now, if you are familiar with web developing you must have understood from my post that I don’t know much about how websites work. If you are not familiar with web developing, why even bother? Regardless, thanks a lot for your inputs.
Why? Because I put a lot of time, effort and money into capturing, scrubbing and analyzing the data I publish on my site and I do not take kindly to people trying to take my data and use it for commercial purposes, without permission and an agreement.

You would not take someone’s car for a drive without asking, so why would you think of taking their data without asking?
Reply With Quote
This user would like to thank Jim2007 for this useful post:
  #14  
Old 10.05.2021, 09:16
Phil_MCR's Avatar
Forum Legend
 
Join Date: Oct 2009
Location: Basel
Posts: 14,554
Groaned at 281 Times in 187 Posts
Thanked 18,379 Times in 7,702 Posts
Phil_MCR has a reputation beyond reputePhil_MCR has a reputation beyond reputePhil_MCR has a reputation beyond reputePhil_MCR has a reputation beyond reputePhil_MCR has a reputation beyond reputePhil_MCR has a reputation beyond repute
Re: Any web developer here to give some advice?

Quote:
View Post
Hi Jim2007

Thank you very much for taking the time to answer. I understand and I agree with your points mentioned above. The property valuation case is just an example not the concept I am working on. I also know where to get the data. What I donít know is how to get data from website A and move it to website B. I am going to pay someone to do it for me but I am trying to find out what my options are and how feasible they are.

Now, if you are familiar with web developing you must have understood from my post that I donít know much about how websites work. If you are not familiar with web developing, why even bother? Regardless, thanks a lot for your inputs.
I know someone who created a webscraping bot to pick up housing data. They initially got into legal trouble and then were technically blocked. They found a pretty clever way to do this eventually.

Anyway my main advice is: if you are not a technical person that will make the website yourself: do not go into a business that is a website business (i.e. not just a webshop). I've seen too many people sink tons of money into their new 'business' that I could have told them will fail right at the start.

Apart from going into an area with no moat and global competition with zero edge - you simply don't know what you don't know that is required to succeed.
__________________
By replying to this post, you hereby grant Phil_MCR a royalty-free license to use, in any way, anything posted by you on the internet. If you do not accept, stop using EF and delete your account.
Reply With Quote
This user would like to thank Phil_MCR for this useful post:
  #15  
Old 07.05.2021, 22:23
Member
 
Join Date: Feb 2020
Location: Zurich
Posts: 156
Groaned at 20 Times in 9 Posts
Thanked 143 Times in 79 Posts
spark is considered knowledgeablespark is considered knowledgeablespark is considered knowledgeable
Re: Any web developer here to give some advice?

Quote:
View Post
An example: a property valuation website where someone inputs the data of a property ( size, bedrooms, balkony, garden, year of construction, etc..) and the website provides an objective price of that property. This is where my questions start:

1. Does the website already have a database of property prices so it takes the inputs of the user, compares it with the database and provides the average price of similar properties? If this is the case, I assume the database needs to be updated frequently in order to reflect current market prices, right?
I'd imagine you would need to build and periodically update a statistical model of prices in each area and just run it when a user query comes in. That's what IAZI / ZKB do, every quarter they update their models, and for some $$$ they will run it a few times for you on your parameters and give some generic statistical reports for your chosen postcode.

Beware current offers on the market != transaction prices. Houses can go for both above or below. In hot areas like Zurich, Zug there's a shortage of supply right now and houses sell for insane prices well above asking

In general. a big problem with swiss real estate market is that transaction prices are not public. Big banks know the prices because they are involved in the deals, and you're not - you will have to resort to scraping homegate data and guesstimating prices from it. That data will be always of inferior quality. Until you can partner up with one of the big boys who has real transaction data.



Quote:
View Post
An example: a property valuation website where
2. Or, does the website somehow search live and dynamically the internet for properties being sold (e.g homegate, immoscout24) and provide with an average price based on what similar properties are on the popular property sales website?
If you partner up with them and they give you API to access their data, sure, you could do it this way. But like others said, unlikely to happen unless you can come with a compelling value proposition to incentivize them to open the data.


You might have better luck asking the big banks - I guess they license their price models to smaller banks for mortgage purposes, they might as well license to you. For $$$$$.
Reply With Quote
This user would like to thank spark for this useful post:
  #16  
Old 07.05.2021, 23:51
Murloc's Avatar
Forum Veteran
 
Join Date: Aug 2010
Location: Bern
Posts: 512
Groaned at 5 Times in 5 Posts
Thanked 568 Times in 285 Posts
Murloc has an excellent reputationMurloc has an excellent reputationMurloc has an excellent reputationMurloc has an excellent reputation
Re: Any web developer here to give some advice?

if you're scraping other people's data, you will need lots of maintenance. E.g. if a website wants users but does not want abusers and copy cats, they will try to make it difficult. At a minimum, they will routinely update their interface because of other reasons.
Free data will also not have the best quality.

If you are paying to access data (typically through APIs), you'll still need maintenance but it will be more predictable and you have technical support.

Besides this, everything is open as to how you interpret the data and use it to make a new estimation based on user inputs (which will still not consider everything...).

If you have an idea yourself as to how you want to model this stuff, imho you can probably also learn enough programming to get a working demo. Converting it in a web service is something that can come later.
Reply With Quote
The following 2 users would like to thank Murloc for this useful post:
Reply




Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Hello stranger! Would you give me some advice? longterminvestment Introductions 6 30.07.2018 15:05
Could you give me some advice on which CARD to get? KiaCF Finance/banking/taxation 9 10.12.2014 14:50
Weird advice that "technical people" give in shops give BaselLife TV/internet/telephone 25 15.01.2013 10:50
Give me some job advice, please Rebells18 Employment 2 26.04.2012 11:25
Business Intelligence Developer (IBM Cognos Developer) - salary in Zurich monte82 Employment 1 23.02.2012 13:15


All times are GMT +2. The time now is 00:19.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.
LinkBacks Enabled by vBSEO 3.1.0