使用java web将商品的数据

  • 时间:
  • 浏览:833
  • 来源:成都艾邦软件开发

1.首先要排查淘宝获取商品数据的请求方法
输入淘宝网后打开开发者工具F12点击网络。我用的是火狐浏览器

2.爬取数据首先需要获取如下请求头参数
cookie
Referer
3.获取到基本信息后替换如下java代码中的对应位置代码如下

import java.io.ByteArrayOutputStream;import java.io.InputStream;import java.net.HttpURLConnection;import java.net.URL;public class taobao {\tpublic static void main(String[] args) {\t\ttry {String url  /search?q%E9%A5%BC%E5%B9%B2imgfilecommendallssids5-esearch_typeitemsourceIdtb.indexspma21bo.2017.201856-taobao-item.1ieutf8initiative_idtbindexz_20170306;URL realUrl  new URL(url);HttpURLConnection connection  (HttpURLConnection) realUrl.openConnection();connection.setRequestProperty(accept, */*);connection.setRequestProperty(connection, Keep-Alive);connection.setRequestProperty(Referer, /search?q%E9%A5%BC%E5%B9%B2imgfilecommendallssids5-esearch_typeitemsourceIdtb.indexspma21bo.2017.201856-taobao-item.1ieutf8initiative_idtbindexz_20170306);connection.setRequestProperty(user-agent, Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0);connection.setRequestProperty(Cookie, isgBKSkHbPtmFr5u-wWzetE4qH9dqKWPcinA1Zw6L7EtW8yaUcz4Ek0N7OLKUHxqgD_; leBIUYKLVjRUD9zzEBO5Cnurza77TzIOV1kPzaNbMiInca6NC1FZm8NCQO6rkRdtfgtfXHFxyyIVnedeDy64dg2HvCbH_7qCkixJ6-; tfstkcBiVBPgX1nK2IgrLmoZN1SAldg2Aa_DmIgyQo40dN8ZvHBUu_sYR68XEN8y1WhUc.; cnaRoBIFSzrQFoCAdOQevvZwOn8; miid518745632110493350; thwcn; UM_distinctid17608fd033d2db-057cbf7fac634f8-4c3f2779-1fa400-17608fd033e412; td31becaca80987bb75837c77484d2d98; _m_h5_tke7afd812cbf21125ef802b63c8df94c6_1615710111436; _m_h5_tk_encfd3a73ffa66e2b2242…e390a0d829; existShopMTYxNTcwMDg4Nw%3D%3D; uc4id40%40U2LDagxTFgkHppc9r9oGtNL7iNUpnk40%40oijaZ6LWXvj5TA1rnr%2B24BPd4g%3D%3D; tracknick%5Cu7F8E%5Cu4E3E%5Cu7684%5Cu4E8C; _cc_Vq8l%2BKCLiw%3D%3D; encZxiYLYwbwjFjE%2F9z8Ry2Ak9smH38MLEK%2Fw6nAKoUZE5RDeYeor0cyWVYpncyvKZHSGpS3MKKCsx6hyn%2FNoJqfg%3D%3D; JSESSIONID250725228E4A009D0FBF207D59D02264; hngCN%7Czh-CN%7CCNY%7C156; mtci0_1; uc1pas0cookie16VT5L2FSpNgq6fDudInPRgavC%2BQ%3D%3Dcookie14Uoe1hx5b4Yrf5A%3D%3Dcookie21Vq8l%2BKCLjA%2BlexistShopfalse);// 建立实际的连接connection.connect();//请求成功System.out.println(请求状态connection.getResponseCode());//if (connection.getResponseCode()  200) {InputStream is  connection.getInputStream();ByteArrayOutputStream baos  new ByteArrayOutputStream();//10MB的缓存byte[] buffer  new byte[10485760];int len  0;while ((len  is.read(buffer)) ! -1) {baos.write(buffer, 0, len);}String jsonString  baos.toString();System.out.println(jsonString:jsonString);baos.close();is.close();//转换成json数据处理//}\t} catch (Exception e) {\t\tSystem.out.println(e);\t}}}

4.执行数据如下。获取的是整个页面的数据。商品数据需要自己截取出来做处理

一.目的

在jsp页面中显示手机商品的信息。

二.思想步骤

2.1收集手机图片信息价格信息手机名信息等。

2.2 创建mysql数据库新建表将信息存入表中。

2.4 创建三个类DataSourceUtilsproductproduct_servlet

2.4.1 DataSourceUtils数据库连接池

2.4.2 product 存储获取到商品信息

2.4.3 product_servlet 连接数据库获取信息并将product对象存在List集合中再将集合设置在requset域中转发到jsp页面。

2.5 创建jsp页面 获取request域中的List集合遍历集合通过jsp脚本代码将信息显示在页面上。

三.实际步骤

3.1 mysql数据

3.1.1 创建数据库 数据库名:product创建新表表名message.

3.1.2 将数据存到message表中  手机信息百度云/s/1fMKzWmpuO3FTiRHjhn4Nkg 提取码xrc8。

   3.1.3 存储成功

   3.2 java代码

 3.2.1 DataSourceUtils 类

注意此处需要导入jar包并且需要将修改好了的c3p0-config.xml 配置文件放在web项目的src目录下

  /s/1eev43QR5TAFKi_ZRqIn_PQ   提取码ebcw此处是jar包和xml配置文件

package cn.jsp.getdate;import java.sql.Connection;import java.sql.ResultSet;import java.sql.SQLException;import java.sql.Statement;import javax.sql.DataSource;import com.mchange.v2.c3p0.ComboPooledDataSource;public class DataSourceUtils {private static DataSource dataSource new ComboPooledDataSource();private static ThreadLocalConnection tl new ThreadLocalConnection();// 直接可以获取一个连接池public static DataSource getDataSource() {return dataSource;}// 获取连接对象public static Connection getConnection() throws SQLException {Connection con tl.get();if (con null) {con dataSource.getConnection();tl.set(con);}return con;}// 开启事务public static void startTransaction() throws SQLException {Connection con getConnection();if (con ! null) {con.setAutoCommit(false);}}// 事务回滚public static void rollback() throws SQLException {Connection con getConnection();if (con ! null) {con.rollback();}}// 提交并且 关闭资源及从ThreadLocall中释放public static void commitAndRelease() throws SQLException {Connection con getConnection();if (con ! null) {con.commit(); // 事务提交con.close();// 关闭资源tl.remove();// 从线程绑定中移除}}// 关闭资源方法public static void closeConnection() throws SQLException {Connection con getConnection();if (con ! null) {con.close();}}public static void closeStatement(Statement st) throws SQLException {if (st ! null) {st.close();}}public static void closeResultSet(ResultSet rs) throws SQLException {if (rs ! null) {rs.close();}}}

3.2.2 product 类JavaBean)

package cn.jsp.getdate;public class product {int id;String product_name;String product_img;double product_money;public int getId() {return id;}public void setId(int id) {this.id id;}public String getProduct_name() {return product_name;}public void setProduct_name(String product_name) {this.product_name product_name;}public String getProduct_img() {return product_img;}public void setProduct_img(String product_img) {this.product_img product_img;}public double getProduct_money() {return product_money;}public void setProduct_money(double product_money) {this.product_money product_money;}}

3.2.3  product_servlet 类

package cn.jsp.getdate;import org.apache.commons.dbutils.QueryRunner;import org.apache.commons.dbutils.handlers.BeanListHandler;import javax.servlet.ServletException;import javax.servlet.http.HttpServlet;import javax.servlet.http.HttpServletRequest;import javax.servlet.http.HttpServletResponse;import java.io.IOException;import java.sql.SQLException;import java.util.List;public class product_servlet extends HttpServlet {protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {// 连接数据库编写sql语句获取数据。QueryRunner runnernew QueryRunner(DataSourceUtils.getDataSource());String sqlselect * from message;//将数据存在List集合中Listproduct productsnull;try {products runner.query(sql,new BeanListHandlerproduct(product.class));}catch (SQLException e){e.printStackTrace();}//将list集合存在request域中request.setAttribute(productsList,products);//转发到show_product.jsp 页面进行显示request.getRequestDispatcher(/show_product.jsp).forward(request,response);}protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {this.doPost(request,response);}}

3.2.4 jsp页面

% page importjava.util.List %% page importcn.jsp.getdate.product %%-- Created by IntelliJ IDEA. User: Administrator Date: 2019/3/19 Time: 19:39 To change this template use File | Settings | File Templates.--%% page contentTypetext/html;charsetUTF-8 languagejava %htmlheadtitleShow_Product/titlelink relstylesheet typetext/css hrefcss/show_Product.css//headbody!--导航栏--div idpagediv idAppBar!--logo--div classlogoSecond-hand mobilebr/phone/divulli首页/lili商店/liliapp下载/lili联系我们/li/ul/div!-- 显示商品--div idcommodity%//System.out.println(request.getAttribute(productsList));Listproduct products(Listproduct)request.getAttribute(productsList);if(products!null){%div classlow%for (product product:products){%ulli classimgimg src%product.getProduct_img()% classmobile_phone//lili classname% product.getProduct_name()%/lili classmoney% product.getProduct_money()%/lili classaffirm购买/li/ul%}%/div%}%/div/div/body/html

3.2.5 jsp 的css样式

*{margin: 0px;padding: 0px;}#page{width: 100%; }#AppBar{width: 100%;height: 50px;background-color: #9278f7;padding-top: 25px;}#page .logo{float: left;width: 20%;height: 55px;background-color: #6b6b6b;margin-left: 25px;margin-top: -25px;text-align: center;font-family: 楷体;font-size: 16px;color: white;padding-top: 20px;opacity: 0.7;}#page #AppBar ul li{width: 10%;height: 45px;float: left;margin-left: 10px;text-align: center;list-style: none;font-size: 15px;color: wheat;}#commodity{width: 100%;background-color: #FFFDFE;}#commodity .low{width: 100%;}#commodity .low ul{margin: 50px;list-style: none;float: right;}#commodity .low .img{width: 150px;height: 150px;border: 1px dotted #333333;}.mobile_phone{width: 100%;height: 100%;}#commodity .low .name{width: 150px;height: 45px;font-family: 宋体;font-size: 10px;text-align: center;}#commodity .low .money{width: 150px;height: 21px;font-family: 宋体;font-size: 10px;text-align: center;color: red;}#commodity .low .affirm{width: 150px;height: 21px;background-color: #07524C;font-size: 12px;color: white;text-align: center;cursor: pointer;}

四.演示

发布项目根据自己的项目输入http://localhost:8080/web_war_exploded/product