Overview of Different Web Structures

Overview of divergent Web StructuresDark Internet, Dark net ( tear manduction), Turbo10, Meta- attempt EngineMansi Iyengar (SCU ID W1170603)This project is targeted for the re assayers to gull insight into the various t dilute structures. The primary focus being full-bodied net and ugliness net.It also throws light on consign communion in regretfulness net along with meta look railway locomotives substance ab character by them.This everyplacelay is based on down(p) net, meta await locomotives and accommodate overlap mechanism. It has been categorized in the form of chapters.Each chapter gives us the beneath informationChapter 1 tells us about the different structures in the sack upChapter 2 pictures an overview of minatory net and TORChapter 3 describes the single point sharing mechanism in smutty net and commonly utilise sexual climax pathesChapter 4 focuses on operation of meta chase engineChapter 5 talks about Turbo10 take c atomic number 18 eng ine for buddy-buddy netThe joinage of to each unitary web page is referred by websites structure. conceive for a website having high gear number of web pages. In such scenario, crawlers should put on the ease to find the subpages. exercise 1 Web structure 1 invention 1 portrays entire web structure that comprises ofSurface web tardily webDark webSurface webThe traditionalistic network we use for everyday life is the world wide web.Represented by www.Deep webDeep web provides the out of sight procedure of the internet. Search engine do non index these. Thus stocky web stands apart from the traditional web. Some comp adeptnts of deep web accommodate email, online banking, on demand videosDark webDark web is different from supra traditional web types.Dark net stands for a network admittanceed via proper(postnominal) softw ar or configuration employ protocol that atomic number 18 not standardized.For everyday life, we use regular see engines such as google. The exi sts websites that set upnot be looked up on google. For such websites need a particular operation in govern to access. Such websites be known as unrelenting web.Dark net comprises of de centralized computers that collaborate in tandem to air in coiffeion from origin to the target. Some of the popular biased net be Limewire, Gnucleus. Users ar parts that sh are information. Hosts are the computers that second share the information.Dark web is also known as blackened net or no-count internet.For ease of understanding we bear cerebrate software denominationd as Tor. Tor provides the functionality wherein the exploiter derrierenot be identified on traditional websites.TORTor also provides sites that have hidden usefulnesss. Basically the hidden or anonymous feature is leveraged to masquerade where inn advanceer is situated. Tor makes it al virtually impossible to trace the members accessing website.Tor is not fool proof and comprises of many flaws. These pertain to security configuration. This gap can be used by officials to determine the real server location. The popularity of TOR is extremely high and hence most commonly used. Hence the focus on this example.Features of dark netHowever, dark nets provide additional facilities different from preceding(prenominal). These may embroil collaborative cause of handling non-licensed software and contentContent free-base on dark netOn two fronts dark net facilitates anonymity member visiting website and also for website itself. Governments too use dark net. For example, in order to gain data bypassing and censorship, Tor provides that electionDark net too can be used by politicians. This is serves as a platform to negotiate and take decision. Secrets are revealed by whistle blowers on certain sites so that they can avoid being traced. Webpages such as Strongbox sponsor serve such purpose.This leads to establishment of websites that traditionally are not used. They can be used for anti-social purpose. For example, stolen credit card information, illegal currency, drugs and weapons are found here. Gambling activities too are provided.Illegal drugs and goods, part of commercial dark market are made popular by silk road and blame market. These are attacked by legal teams.Alternate hacking services are sold. These are at group level or individualistic level. This has led to cyber-crimes and predict cyber investigation. Government has started looking into these victimization tools procured from Procedia Computer Science journal. Denial of service attacks too have been made through Dark Web. 4File SharingFile sharing involves the method of dividing or enabling digital media access such as audio files, video files, programs or eBooks.There are bigeminal ways to attain file sharing. Generally used approaches of storage, transmittance include web url hyperlinked docs., p2p networks and centralized servers. 3Figure 2 file sharingThe above go into shows us how figure shows ho w file sharing can be achieved between access controlled digital establishments like hidden, worldly concern and invited.P2P file sharing figure outComputers or nodes are connected with each separate as part of P2P network. The members have the ability to share or download information immediately via the web.Figure 3 comrade to Peer networkAbove figure shows peer to peer network. Based on the model of self-server and client model, P2P is set up in general.for example, when the member connects to the P2P client and after initiating file download, he gets connected with other members downloading the information. In other words, others peers serve as file server. The initial member also acts as file server when another peer starts to download the file. Eventually this leads to augmenting of download speed. Some commonly used deluges for P2P are bit cloudburst and cloudburst. Other popular P2P networks being BearShare and LimeWire. 2OnionShareConsider App called OnionShare that is available for multiple Operating Systems. This is a P2P program. Entire information from TOR anonymity network is sent through this file sharing program.consider regular file sharing system needs faith. Law regulators can splatter into these.Tor helps bypass third party, there is direct sharing of file from one somebody to another via anonymous networkFigure 4 OnionShareThe above figure describes OnionShare demonstrate. After Tor instance is passning, OnionShare will run. The beginning part is by starting a local web server that is seeable as tor hidden service, generating web link that cannot be guessed out. This web link is used to download the file.The web link is then provided to the recipient person via secure methodthe webserver is cancelled The moment the receipt downloads the file,Now the file becomes inaccessible to anyone. These is scope to expand this further by having a round-the-clock running server in order to share with multiple recipients 5BitTorrent File S haring ProcessBackgroundBitTorrent file sharing protocol for dark net. BitTorrent is a peer-to-peer file sharing protocol and the most popular one according to a reportby (2008). In the said(prenominal) report one can read that BitTorrent takesup a substantial measuring rod of all the internet traffic in the world, between 27 and 55 percent depending on geographical location. This makes BitTorrent not only the most used protocol for file sharing, but of all application spirit level protocols.As per Schulse and Mochalski , Bit Torrent is popular file sharing protocol. This is also used by Darknet.BitTorrent breaks data into parallel small chunks instead of single chunk, enhancing reliableness and reducing vulnerabilityApproaches for bit torrent file sharingProvided below are two approaches for bit torrent file sharingTorrent file approachTFigure 5 Torrent file sharing approachA torrent file is setup that has tracker, file contents. This file is littler than original file. Seed is the uploader and leech is the downloader.Above figure describes the torrent file sharing approach. User first downloads the torrent. From torrent the set of peers is found. From the network, file pieces are exchanged. Files are treated instead of atomic blocks but as smaller chunks.Advantage is that the bandwidth is spread among peer as against just the seeds.The tracker for a torrent provides peers a peer list. In this manner, bit torrent dialogue takes place, which is the similar process involved for seeders and leachesDTH approachThere is a second approach that bypasses the tracker. This provides greater anonymity. This approach is performed without generating torrent file.For this purpose, bit torrent protocol ends up implementing DHT i.e. distributed hash table.Figure 6 DTH file sharing approachThe above figure describes DTH file sharing approach. DHT provides a set of key, value salve in it. The working is similar to how has table works. Nodes have ability to watch noes b ased on the key. There is unique identification for each node. For this the communication utilized is User Datagram Protocol (UDP).The benefit is that there no centralized trackers. However, the track list is provided by peers that send peer list. Traditional torrent file has web link to a tracker. This step is by passed in second approach, wherein DHT web link is used. DHT network is entered into by the peer using the URL. From the entry node, it can essay across other networks for files and peer liaison for file sharing. 6The payoffs of other databases are combined in meta hunt engine. It takes up the concatenation of different results from various chase engines and links then to various resources. 1Virtual databases are generated from the mirrors of the physical database results of other search engines. These virtual databases are generated from the meta search engines. The searches are concentrated over certain criteria. 2Background trenchant the web with multiple search en gines was the issue tackled by researchers. One of the outputs was meta search engine. Search Savvy was the first engine discovered and used by Daniel Dreilinger from Colorado State University. This engine looks over the results from 20 different directories and search engines at once. Meta search engine crawler discovered by Eric Selberg at Washington University. It used its own search engine syntax and searched over 6 search engines.Dogpile formed at University of Pittsburgh and Pennsylvania State University metrical ranking and overlap of various search engines. This proved to have one of the best results. 3AdvantagesMore information and search coverage can be achieved by using meta search engines. This helps the user to get faster and ideal results. The research of the user results is generated in unique ways by using the indexes which are aggregated by search engines. The input effort for a meta search engine and normal search engine are the same(p) but the results retrieved are more. They also reduce the effort of users to type on various search engines to find file and resources.DisadvantagesTranslation of the interrogatory format or understanding the query forms can be an issue in the meta search engine. All the results sets of a query are not addicted to a user as the links generated from a meta search engine are particular(a). The links generated are limited. Promoted websites are tending(p) higher anteriority over other websites.It is probable that the user will get multiple results from the search engine. It may not be stream lined. This is especially an issue due to more coverage over the queried topic. The users find it difficult to use meta search. They might not be able to get any particular information.OperationThe above figure describes the meta search operation. The search engine receives a query from the user.The key parts are as below negociate The query needs a pre-processing as each search engine has a specific format of the dat a being fed in. As they depend on different operators and they do not share the same syntax. The output is a series of ordered set documents.Rank the documents are raked post identifying the result pages and also in the order of the most pertinent links. 4The input query is passed to the search engines database. It creates a virtual database. This helps to combine data from various sources. Duplicates can be generated as various search-engines have different methods of indexing the data. This output data is then processed by the meta-search engine. The rewrite list is produced for the user. They would respond in the following ways.1) Access to the meta-search engine credit entry is provided which also includes the private access to the database system The changes made on the database system would be captured.2) In a non-cooperative way, the access to the interface may or may not be provided.3) Meta search engine is denied access to the search engine.Architecture of rankingAbove f igure describes the architecture of the meta search engine. It consists of the users query being sent to the meta search engine. It contacts the other search engines its connected to. They process the results and generate resources. These results are then preprocessed and then given back to the meta search engine as a response. This collective response is given to the user. federationFusion is a process for data filtering. It helps to build up effectual results. assembly FusionUnrelated data is indexed via search engines it is dealt by the Collection Fusion process which is also a distributed retrieval process. Ranking on the data is based on how probable is the data to give the need information to the user. It picks up the best resources and ranks it. The resources which are take aimed are combined in a list and given to the user.Data FusionFigure 9 Data Fusion ArchitectureAbove figure give the architectural reference of the Data Fusion.L0 The user information which the sensors and the sources process is given to the conglutination algorithmL1 The machine process of converting L0 output and it feeds it to the L1 processL2 L3 The human process of manually marking the retrieved data as relevant or notL4 Resource management layer adds to the sources and the L2/L3 layer providing th necessary feedbackL6 The billing management helps to add context to the user query. This is added to the resource management layerL5 Once all the processing through the layers is commit the data is given back to the user.This is fusion mechanism which relies on the common data sets which are retrieved by search engines. The initial ranks are combined into a list. The depth psychology of the actual ranks of the documents retrieved is calculated. The links of the documents with the highest relevancy score is selected. ComboSum is one of the techniques used to normalise the scores. The scores produced are incomparable as different search engines run on different algorithms.Example s of Meta-Search EnginesDeepPeepFigure 10 DeepPeep SnippetAbove figure is a snippet for DeepPeet. Public web pages are indexes and expanded by search engine. It does so by search through various exoteric databases. Traditional search engines cannot index by DeepPeep. It also looks to find urls into deep web.IncyWincyFigure 11 Incywincy snippetAbove image is Incywincy snippet. It uses many search engines, combines and filters the results. It crawlers more than 200 million pages. It uses its unique algorithm for relevance tests. eat upPlanetFigure 12 Complete PlanetAbove figure is snippet of Complete Planet. It indexes and crawlers many search engines and special databases which are not public. It is one of the principal(prenominal) search engines of the deep web.Other Deep Web search engines include Intute, Infomine, Scirus, Turbo 10.The above figure is the Turbo 10 is a meta search engine snippet. It helps to plan a universal reference for Deep Net. It refers to topic specific se arch engines. Google and Yahoo like crawler-based search engines do not use these files. Turbo 10 also lets you add more engines to the collection on which your search query is executed. 7ProsThe positive thing about turbo10 is that it has the ability to connect and maintain continuative to online databases in bulk.This provides connecting to thousand engines in a fully automated cogency which is scalable to further connect to another thousand.Turbo10 also finds content on deep net. Deep net encompasses multiple databases covering wide deviate of topics such as business, colleges, government teams. These are not accessible to traditional web engines and google. Traditional search engines are helpful for indexing nonmoving pages.Thus a rich experience is provided as we can tap into multiple databases across such a turgid range of domains, empowering the user.Turbo provides the option of having ten search engines to search from.Unlike other search engine like AskJeeves, that face s drawback of segregating information into different boxes, turbo streamlines the process by proving the result in weighted listing. This makes Turbo10 the ultimate search engine of search engine.Consider the feature that say four search engines are selected, turbo10 does the work of selecting additional half a dozen search engines as it deems as best fit.The result set would be based on either relevance or speed. The decision is nonparasitic of the search independent of search engine selected.The above reduction of search result occurs by usage of clusters in box on left hand side. The benefit of having clusters a few time, desired result set is obtained faster as against advanced searches or logical expressionTurbo10 provides a bleak search paradigm. For a given page, there are ten result lists on a page. These have arrows. Thought this may also be unpopular, this generates hundred results for a given search.The average case being cardinal to forty result set being provided as three or four pages are returned. . Google gives lots of options. Unlike Google, Turbo10 simply limits the search results and provides limited result setThe main idea is that with Turbo 10, ability to choose search enigne is provided. Whereas say for a particular item through google, we need to keep searching and may find the same at pat a latter page say page fortyHow is the ability provided to select ten search engines?The task is completed via the web. A collection name is assigned to the ten search engines, mapped to your email.CapabilityTurbo10 has a vision to bulge as a leader for search enginePlan to use amazon based recommendation algorithms for personalized searching. This would include personalized indite search, providing bookmarking feature an ecosystem to be setup wherein the users profile would interact with other users. extra browsing options would be generated for users. Revenue model would tap into sponsored web links. These would be flagged licensing of the product to corporates.ConsTturbo10 faces the drawback of being highly cluttered. Google has lots of white spaces. The search results are not cramped up. Turbo10 is not visually appealing. It has purple colour concealment that may not be appealing to allMany multiplication the returned result set is cluttered. Sometimes the result set gets jumbled modify result set. Additionally, there is no consideration for punctuation marks or logical expressions. Does not have ability to cache information. Due to high demand and expectation, knocked offline.Intense competition with google.The project is based on dark net ecosystem comprising of file sharing mechanisms and meta search engine. Dark net has unendingly using evolving engineering science such as TOR, OnionShare. These help enable provide benefits of being anonymous protecting their identity. Similarly, the website to gets to preserve their anonymity. Dark net serves as a platform of communication to mediate, take decision and shar e information. These are used by different agencies including government and non-traditional activities such as whistleblowing.Products that are not listed in traditional websites too are available on such platforms.File Sharing approaches save bandwidth by helping sharing information efficiently. Easier to back up information. Fault tolerance is enhanced through decentralized approach part of peer to peer networking. Comparatively there is ease of maintenance over traditional files sharing system.Meta search engines help to get streamlined results from various search engines which helps to improve relevancy of searches.The benefits of dark net also include cross border payments, ensure complete privacy of sender and receiver. There is application in cryptocurrency, digital trading, eliminating middlemen. amaze up of hassle free payments independent of weekend or holidays. realise of decentralization is control over your content that you want to share. Other advantages include est ablish net neutrality which means that internet can be used for all and not monopolized.Hence there is a large scope for expansion in dark net space and technology is playing a pivotal role in enhancing its adoption.1 https//www.linkedin.com/pulse/internet-deep-web-dark-net-firas-saras2 https//sysinfotools.com/ intercommunicate/peer-to-peer-file-sharing/3 http//www.spiroprojects.com/blog/cat-view-more.php?blogname=What-is-file-sharing-system?id=2624 https//www.howtogeek.com/275875/what-is-the-dark-web/5 https//darkwebnews.com/anonymity/some-ways-to-share-files-anonymously/6 Johan Andersson Gabriel Ledung, Darknet file sharing application of a private peer-to-peer distributed file system concept7 http//techdiction.blogspot.com/2007/01/turbo-10-search-deep-net.htmlURL Uniform Resource locaterTORThe Onion RouterUDPUser Datagram ProtocolDTHdistributed hash tableWWWWorldwide web