Browse Source

Added WebLoader

master
ThePBone 10 months ago
parent
commit
a9123618fb
17 changed files with 2330 additions and 1 deletions
  1. +0
    -1
      WebLoader
  2. +31
    -0
      WebLoader/.gitignore
  3. +165
    -0
      WebLoader/LICENSE
  4. +79
    -0
      WebLoader/README.md
  5. +22
    -0
      WebLoader/WebLoader.pro
  6. +228
    -0
      WebLoader/src/HttpMultiPart_p.cpp
  7. +84
    -0
      WebLoader/src/HttpMultiPart_p.h
  8. +186
    -0
      WebLoader/src/NetworkQueue_p.cpp
  9. +128
    -0
      WebLoader/src/NetworkQueue_p.h
  10. +207
    -0
      WebLoader/src/NetworkRequest.cpp
  11. +178
    -0
      WebLoader/src/NetworkRequest.h
  12. +109
    -0
      WebLoader/src/NetworkRequestLoader.h
  13. +74
    -0
      WebLoader/src/NetworkRequestPrivate_p.h
  14. +340
    -0
      WebLoader/src/WebLoader_p.cpp
  15. +164
    -0
      WebLoader/src/WebLoader_p.h
  16. +212
    -0
      WebLoader/src/WebRequest_p.cpp
  17. +123
    -0
      WebLoader/src/WebRequest_p.h

+ 0
- 1
WebLoader

@@ -1 +0,0 @@
Subproject commit 52ce73ff0900c2fb7d481996cb6dd219b4560836

+ 31
- 0
WebLoader/.gitignore View File

@@ -0,0 +1,31 @@
# Compiled Object files
*.slo
*.lo
*.o
*.obj

# Precompiled Headers
*.gch
*.pch

# Compiled Dynamic libraries
*.so
*.dylib
*.dll

# Fortran module files
*.mod
*.smod

# Compiled Static libraries
*.lai
*.la
*.a
*.lib

# Executables
*.exe
*.out
*.app

WebLoader.pro.user

+ 165
- 0
WebLoader/LICENSE View File

@@ -0,0 +1,165 @@
GNU LESSER GENERAL PUBLIC LICENSE
Version 3, 29 June 2007

Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.


This version of the GNU Lesser General Public License incorporates
the terms and conditions of version 3 of the GNU General Public
License, supplemented by the additional permissions listed below.

0. Additional Definitions.

As used herein, "this License" refers to version 3 of the GNU Lesser
General Public License, and the "GNU GPL" refers to version 3 of the GNU
General Public License.

"The Library" refers to a covered work governed by this License,
other than an Application or a Combined Work as defined below.

An "Application" is any work that makes use of an interface provided
by the Library, but which is not otherwise based on the Library.
Defining a subclass of a class defined by the Library is deemed a mode
of using an interface provided by the Library.

A "Combined Work" is a work produced by combining or linking an
Application with the Library. The particular version of the Library
with which the Combined Work was made is also called the "Linked
Version".

The "Minimal Corresponding Source" for a Combined Work means the
Corresponding Source for the Combined Work, excluding any source code
for portions of the Combined Work that, considered in isolation, are
based on the Application, and not on the Linked Version.

The "Corresponding Application Code" for a Combined Work means the
object code and/or source code for the Application, including any data
and utility programs needed for reproducing the Combined Work from the
Application, but excluding the System Libraries of the Combined Work.

1. Exception to Section 3 of the GNU GPL.

You may convey a covered work under sections 3 and 4 of this License
without being bound by section 3 of the GNU GPL.

2. Conveying Modified Versions.

If you modify a copy of the Library, and, in your modifications, a
facility refers to a function or data to be supplied by an Application
that uses the facility (other than as an argument passed when the
facility is invoked), then you may convey a copy of the modified
version:

a) under this License, provided that you make a good faith effort to
ensure that, in the event an Application does not supply the
function or data, the facility still operates, and performs
whatever part of its purpose remains meaningful, or

b) under the GNU GPL, with none of the additional permissions of
this License applicable to that copy.

3. Object Code Incorporating Material from Library Header Files.

The object code form of an Application may incorporate material from
a header file that is part of the Library. You may convey such object
code under terms of your choice, provided that, if the incorporated
material is not limited to numerical parameters, data structure
layouts and accessors, or small macros, inline functions and templates
(ten or fewer lines in length), you do both of the following:

a) Give prominent notice with each copy of the object code that the
Library is used in it and that the Library and its use are
covered by this License.

b) Accompany the object code with a copy of the GNU GPL and this license
document.

4. Combined Works.

You may convey a Combined Work under terms of your choice that,
taken together, effectively do not restrict modification of the
portions of the Library contained in the Combined Work and reverse
engineering for debugging such modifications, if you also do each of
the following:

a) Give prominent notice with each copy of the Combined Work that
the Library is used in it and that the Library and its use are
covered by this License.

b) Accompany the Combined Work with a copy of the GNU GPL and this license
document.

c) For a Combined Work that displays copyright notices during
execution, include the copyright notice for the Library among
these notices, as well as a reference directing the user to the
copies of the GNU GPL and this license document.

d) Do one of the following:

0) Convey the Minimal Corresponding Source under the terms of this
License, and the Corresponding Application Code in a form
suitable for, and under terms that permit, the user to
recombine or relink the Application with a modified version of
the Linked Version to produce a modified Combined Work, in the
manner specified by section 6 of the GNU GPL for conveying
Corresponding Source.

1) Use a suitable shared library mechanism for linking with the
Library. A suitable mechanism is one that (a) uses at run time
a copy of the Library already present on the user's computer
system, and (b) will operate properly with a modified version
of the Library that is interface-compatible with the Linked
Version.

e) Provide Installation Information, but only if you would otherwise
be required to provide such information under section 6 of the
GNU GPL, and only to the extent that such information is
necessary to install and execute a modified version of the
Combined Work produced by recombining or relinking the
Application with a modified version of the Linked Version. (If
you use option 4d0, the Installation Information must accompany
the Minimal Corresponding Source and Corresponding Application
Code. If you use option 4d1, you must provide the Installation
Information in the manner specified by section 6 of the GNU GPL
for conveying Corresponding Source.)

5. Combined Libraries.

You may place library facilities that are a work based on the
Library side by side in a single library together with other library
facilities that are not Applications and are not covered by this
License, and convey such a combined library under terms of your
choice, if you do both of the following:

a) Accompany the combined library with a copy of the same work based
on the Library, uncombined with any other library facilities,
conveyed under the terms of this License.

b) Give prominent notice with the combined library that part of it
is a work based on the Library, and explaining where to find the
accompanying uncombined form of the same work.

6. Revised Versions of the GNU Lesser General Public License.

The Free Software Foundation may publish revised and/or new versions
of the GNU Lesser General Public License from time to time. Such new
versions will be similar in spirit to the present version, but may
differ in detail to address new problems or concerns.

Each version is given a distinguishing version number. If the
Library as you received it specifies that a certain numbered version
of the GNU Lesser General Public License "or any later version"
applies to it, you have the option of following the terms and
conditions either of that published version or of any later version
published by the Free Software Foundation. If the Library as you
received it does not specify a version number of the GNU Lesser
General Public License, you may choose any version of the GNU Lesser
General Public License ever published by the Free Software Foundation.

If the Library as you received it specifies that a proxy can decide
whether future versions of the GNU Lesser General Public License shall
apply, that proxy's public statement of acceptance of any version is
permanent authorization for you to choose that version for the
Library.

+ 79
- 0
WebLoader/README.md View File

@@ -0,0 +1,79 @@
# WebLoader
High level wrapper around QNetworkAccessManager for make network communications easy.

## About
Library allow you to load everything from internet (web pages, images, archives etc.). You can choose how data will be loaded: asynchronously or synchronously.

Library build on top of queue of loaders, everyone of which is simple wrapper around QNetworkAccessManager object. It means that you don't need to warn about memory management, mime types detecting or something else, library does it instead of you.

Based on Qt5.

## How to use
Generic class for interacting with web is NetworkRequest. You can send get, post and raw requests. You can manage referrer, cookies, handle loading progress and of course loading errors for every request.

And we have little helper for common tasks. It named NetworkRequestLoader. It provide simple way to user for download data from internet.

#### #include \<NetworkRequestLoader.h\>
Let's see how easy to load web-pages via NetworkRequestLoader.

Load data synchronously.
```c++
const QByteArray data = NetworkRequestLoader::loadSync("https://github.com");
```
Now let's load data in asynchronous way.
```c++
NetworkRequestLoader::loadAsync("https://github.com", [] (const QByteArray& _loadedData) {
qDebug() << "Loaded" << _loadedData.size() << "bytes.";
});
```
If you need to load thousands of web-pages this is not a problem - just load them!:)
```c++
NetworkRequestLoader::loadAsync(thousandsUrls, [] (const QByteArray& _loadedData, const QUrl& _fromUrl) {
qDebug() << "Loaded" << _loadedData.size() << "bytes from" << _fromUrl;
});
```
Don't worry about anything, library makes all work instead of you.

#### #include \<NetworkRequest.h\>
If you need to more complex management of loading process, NetworkRequest helps you.

Let's send some post request.
```c++
NetworkRequest request;
requset.setRequestMethod(NetworkRequest::Post);
request.addRequestAttribute("id", 1893);
request.addRequestAttributeFile("photo", "/home/user/Images/photo.png");
const QByteArray postStatus = request.loadSync("https://site.com/API/v1/savePhoto/");
```
Okey. Now let's send json data to our server.
```c++
NetworkRequest request;
requset.setRequestMethod(NetworkRequest::Post);
request.setRawRequest("{ \"method\": \"getUsers\" }", "application/json");
const QByteArray usersJson = request.loadSync("https://site.com/API/v2/function/");
```
Want to save session cookies? Just sent instance of QNetworkCookieJar class to NetworkRequest.
```c++
QNetworkCookieJar cookies;
NetworkRequest request;
request.setCookieJar(&cookies);
request.loadSync("https://site.com/API/v1/startSession");

// And in next step you can reuse this request or create new NetworkRequest and sent cookies to them

request.loadSync("https://site.com/API/v1/uploadImage");
```
It's really simple, just try!

## Contribution
We really love feedback. If you have ideas for make it better, or find some bugs, or fix some bugs :), or just want to ask question - you welcome!

## Discussion

* https://forum.qt.io/topic/70627/high-level-wrapper-around-qnetworkaccessmanager-for-make-network-communications-easy
* http://www.forum.crossplatform.ru/index.php?showtopic=10833
* http://www.prog.org.ru/topic_30546_0.html

## Credits

Alexey Polushkin @armijo38 and Dimka Novikov @dimkanovikov

+ 22
- 0
WebLoader/WebLoader.pro View File

@@ -0,0 +1,22 @@
TEMPLATE = lib
TARGET = webloader
DEPENDPATH += . src
INCLUDEPATH += . src

CONFIG += c++11

QT += network xml

HEADERS += src/HttpMultiPart_p.h \
src/NetworkQueue_p.h \
src/NetworkRequest.h \
src/NetworkRequestPrivate_p.h \
src/WebLoader_p.h \
src/WebRequest_p.h \
src/NetworkRequestLoader.h

SOURCES += src/HttpMultiPart_p.cpp \
src/WebLoader_p.cpp \
src/WebRequest_p.cpp \
src/NetworkQueue_p.cpp \
src/NetworkRequest.cpp

+ 228
- 0
WebLoader/src/HttpMultiPart_p.cpp View File

@@ -0,0 +1,228 @@
/*
* Copyright (C) 2015 Dimka Novikov, [email protected]
* Copyright (C) 2016 Alexey Polushkin, [email protected]
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 3 of the License, or any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* Full license: http://dimkanovikov.pro/license/LGPLv3
*/

#include "HttpMultiPart_p.h"
#include "QMimeDatabase"
#include <QtCore/QStringList>
#include <QtCore/QFile>


HttpPart::HttpPart(HttpPartType _type) :
m_type( _type )
{

}

HttpPart::HttpPartType HttpPart::type() const
{
return m_type;
}

void HttpPart::setText(const QString& _name, const QString& _value)
{
setName(_name);
setValue(_value);
}

void HttpPart::setFile(const QString& _name, const QString& _filePath)
{
setName(_name);
setFilePath(_filePath);
}

QString HttpPart::name() const
{
return m_name;
}

QString HttpPart::value() const
{
return m_value;
}

QString HttpPart::fileName() const
{
return value();
}

QString HttpPart::filePath() const
{
return m_filePath;
}



void HttpPart::setName(const QString& _name)
{
if (m_name != _name)
m_name = _name;
}

void HttpPart::setValue(const QString& _value)
{
if (m_value != _value)
m_value = _value;
}

void HttpPart::setFileName(const QString& _fileName)
{
setValue(_fileName);
}

void HttpPart::setFilePath(const QString& _filePath)
{
if (m_filePath != _filePath) {
m_filePath = _filePath;
QString fileName = _filePath.split('/').last();
setFileName(fileName);
}
}





HttpMultiPart::HttpMultiPart()
{
}

void HttpMultiPart::setBoundary(const QString& _boundary)
{
if (m_boundary != _boundary) {
m_boundary = _boundary;
}
}

void HttpMultiPart::addPart(const HttpPart& _part)
{
m_parts.append(_part);
}

QByteArray HttpMultiPart::data()
{
QByteArray multiPartData;
foreach ( HttpPart httpPart, parts() ) {
QByteArray partData = makeDataFromPart( httpPart );
multiPartData.append( partData );
}
// Добавление отметки о завершении данных
{
QByteArray endData = makeEndData();
multiPartData.append( endData );
}
return multiPartData;
}

QByteArray HttpMultiPart::makeDataFromPart(const HttpPart& _part)
{
QByteArray partData;
switch (_part.type()) {
case HttpPart::Text: {
partData = makeDataFromTextPart( _part );
break;
}
case HttpPart::File: {
partData = makeDataFromFilePart( _part );
break;
}
}
return partData;
}

QByteArray HttpMultiPart::makeDataFromTextPart(const HttpPart& _part)
{
QByteArray partData;

partData.append("--");
partData.append(boundary());
partData.append(crlf());

partData.append(
QString("Content-Disposition: form-data; name=\"%1\"%3%3%2")
.arg(_part.name(), _part.value(), crlf())
);

partData.append(crlf());

return partData;
}

QByteArray HttpMultiPart::makeDataFromFilePart(const HttpPart& _part)
{
QByteArray partData;

partData.append("--");
partData.append(boundary());
partData.append(crlf());

{
// Определение mime типа файла
QMimeDatabase mimeTypeDetector;
QString contentType = mimeTypeDetector.mimeTypeForFile(_part.filePath()).name();

partData.append(
QString("Content-Disposition: form-data; name=\"%1\"; filename=\"%2\"%4"
"Content-Type: %3%4%4"
)
.arg(_part.name(),
_part.fileName(),
contentType,
crlf())
);

QFile uploadFile(_part.filePath());
uploadFile.open(QIODevice::ReadOnly);
while (!uploadFile.atEnd()) {
QByteArray readed = uploadFile.read(1024);
partData.append(readed);
}
uploadFile.close();
}

partData.append(crlf());
return partData;
}

QByteArray HttpMultiPart::makeEndData()
{
QByteArray partData;

partData.append("--");
partData.append(boundary());
partData.append("--");
partData.append(crlf());

return partData;
}

QString HttpMultiPart::boundary() const
{
return m_boundary;
}

QString HttpMultiPart::crlf() const
{
QString crlf;
crlf = 0x0d;
crlf += 0x0a;
return crlf;
}

QList<HttpPart> HttpMultiPart::parts() const
{
return m_parts;
}

+ 84
- 0
WebLoader/src/HttpMultiPart_p.h View File

@@ -0,0 +1,84 @@
/*
* Copyright (C) 2015 Dimka Novikov, [email protected]
* Copyright (C) 2016 Alexey Polushkin, [email protected]
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 3 of the License, or any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* Full license: http://dimkanovikov.pro/license/LGPLv3
*/

#ifndef HTTPMULTIPART_H
#define HTTPMULTIPART_H

#include <QtCore/QByteArray>
#include <QtCore/QString>
#include <QtCore/QList>

class HttpPart
{
public:
enum HttpPartType {
Text,
File
};

public:
HttpPart(HttpPartType _type = Text);
HttpPartType type() const;
void setText(const QString& _name, const QString& _value);
void setFile(const QString& _name, const QString& _filePath);

public:
QString name() const;
QString value() const;
QString fileName() const;
QString filePath() const;


private:
void setName(const QString& _name);
void setValue(const QString& _value);
void setFileName(const QString& _fileName);
void setFilePath(const QString& _filePath);

private:
HttpPartType m_type;
QString m_name,
m_value,
m_filePath;
};

class HttpMultiPart
{
public:
HttpMultiPart();
void setBoundary(const QString& _boundary);
void addPart(const HttpPart& _part);

QByteArray data();

private:
QByteArray makeDataFromPart(const HttpPart& _part);
QByteArray makeDataFromTextPart(const HttpPart& _part);
QByteArray makeDataFromFilePart(const HttpPart& _part);
QByteArray makeEndData();

private:
QString boundary() const;
QString crlf() const;
QList<HttpPart> parts() const;

private:
QString m_boundary;
QList<HttpPart> m_parts;
};

#endif // HTTPMULTIPART_H

+ 186
- 0
WebLoader/src/NetworkQueue_p.cpp View File

@@ -0,0 +1,186 @@
/*
* Copyright (C) 2016 Alexey Polushkin, [email protected]
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 3 of the License, or any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* Full license: http://dimkanovikov.pro/license/LGPLv3
*/

#include "NetworkQueue_p.h"
#include "WebLoader_p.h"
#include "WebRequest_p.h"
#include "NetworkRequestPrivate_p.h"

NetworkQueue::NetworkQueue()
{
//
// В нужном количестве создадим WebLoader'ы
// И сразу же соединим их со слотом данного класса, обозначающим завершение
//
for (int i = 0; i != qMax(QThread::idealThreadCount(), 4); ++i) {
m_freeLoaders.push_back(new WebLoader(this));
connect(m_freeLoaders.back(), &WebLoader::finished,
this, static_cast<void (NetworkQueue::*)()>(&NetworkQueue::downloadComplete));
}
}

NetworkQueue* NetworkQueue::getInstance() {
static NetworkQueue queue;
return &queue;
}

void NetworkQueue::put(NetworkRequestPrivate* _request) {
//
// Положим в очередь пришедший запрос
//
m_queue.push_back(_request);
m_inQueue.insert(_request);

//
// В случае, если есть свободный WebLoader
// Извлечем пришедший запрос из очереди и начнем выполнять его
//
if (!m_freeLoaders.empty()) {
pop();
}
}

void NetworkQueue::pop() {
//
// Извлечем свободный WebLoader
//
WebLoader* loader = m_freeLoaders.front();
m_freeLoaders.pop_front();

//
// Извлечем первый запрос на обработку
//
NetworkRequestPrivate *request = m_queue.front();
m_queue.pop_front();
m_inQueue.remove(request);

//
// Настроим WebLoader на запрос
//
m_busyLoaders[loader] = request;
setLoaderParams(loader, request);

//
// Соединим сигналы WebLoader'а с сигналами класса запроса
//
connect(loader, static_cast<void (WebLoader::*)(QByteArray, QUrl)>(&WebLoader::downloadComplete),
request, &NetworkRequestPrivate::downloadComplete);
connect(loader, static_cast<void (WebLoader::*)(int, QUrl)>(&WebLoader::uploadProgress),
request, &NetworkRequestPrivate::uploadProgress);
connect(loader, static_cast<void (WebLoader::*)(int, QUrl)>(&WebLoader::downloadProgress),
request, &NetworkRequestPrivate::downloadProgress);
connect(loader, &WebLoader::error, request, &NetworkRequestPrivate::error);
connect(loader, &WebLoader::errorDetails, request, &NetworkRequestPrivate::errorDetails);

//
// Загружаем!
//
loader->loadAsync(request->m_request->urlToLoad(), request->m_request->urlReferer());
}

void NetworkQueue::stop(NetworkRequestPrivate* _internal) {
if (m_inQueue.contains(_internal)) {
//
// Либо запрос еще в очереди
// Тогда его нужно оттуда удалить
//
m_queue.removeAll(_internal);
m_inQueue.remove(_internal);
}
else {
//
// Либо запрос уже обрабатывается
//
for (auto iter = m_busyLoaders.begin(); iter != m_busyLoaders.end(); ++iter) {
//
// Найдем запрос в списке обрабатывающихся
//
if (iter.value() == _internal) {

//
// Отключим все сигналы
// Обязательно сначала отключить сигналы, а затем остановить. Не наоборот!
//
disconnectLoaderRequest(iter.key(), iter.value());

//
// Остановим запрос
//
iter.key()->stop();

iter.value()->done();
//
// Удалим из списка используемых
// К списку свободных WebLoader'ов припишет слот downloadComplete
//
m_busyLoaders.erase(iter);

break;
}
}
}
}

void NetworkQueue::setLoaderParams(WebLoader* _loader, NetworkRequestPrivate* request)
{
_loader->setCookieJar(request->m_cookieJar);
_loader->setRequestMethod(request->m_method);
_loader->setLoadingTimeout(request->m_loadingTimeout);
_loader->setWebRequest(request->m_request);
}

void NetworkQueue::disconnectLoaderRequest(WebLoader* _loader,
NetworkRequestPrivate* _request)
{
disconnect(_loader, static_cast<void (WebLoader::*)(QByteArray, QUrl)>(&WebLoader::downloadComplete),
_request, &NetworkRequestPrivate::downloadComplete);
disconnect(_loader, static_cast<void (WebLoader::*)(int, QUrl)>(&WebLoader::uploadProgress),
_request, &NetworkRequestPrivate::uploadProgress);
disconnect(_loader, static_cast<void (WebLoader::*)(int, QUrl)>(&WebLoader::downloadProgress),
_request, &NetworkRequestPrivate::downloadProgress);
disconnect(_loader, &WebLoader::error, _request, &NetworkRequestPrivate::error);
disconnect(_loader, &WebLoader::errorDetails, _request, &NetworkRequestPrivate::errorDetails);
}

void NetworkQueue::downloadComplete()
{
WebLoader* loader = qobject_cast<WebLoader*>(sender());
if (m_busyLoaders.contains(loader)) {
//
// Если запрос отработал до конца (не был прерван методом stop),
// то необходимо отключить сигналы
// и удалить из списка используемых
//
NetworkRequestPrivate* request = m_busyLoaders[loader];

disconnectLoaderRequest(loader, request);
request->done();

m_busyLoaders.remove(loader);
}

//
// Добавляем WebLoader в список свободных
//
m_freeLoaders.push_back(loader);

//
//Смотрим, надо ли что еще выполнить из очереди
//
if (!m_queue.empty()) {
pop();
}
}

+ 128
- 0
WebLoader/src/NetworkQueue_p.h View File

@@ -0,0 +1,128 @@
/*
* Copyright (C) 2016 Alexey Polushkin, [email protected]
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 3 of the License, or any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* Full license: http://dimkanovikov.pro/license/LGPLv3
*/

#ifndef NETWORKQUEUE_H
#define NETWORKQUEUE_H

#include <QObject>
#include <QSet>
#include <QMap>

class WebLoader;
class NetworkRequestPrivate;

/*!
* \brief Класс, реализующий очередь запросов
* Реализован как паттерн Singleton
*/
class NetworkQueue : public QObject
{
Q_OBJECT
public:

/*!
* \brief Метод, возвращающий указатель на инстанс класса
*/
static NetworkQueue* getInstance();

/*!
* \brief Метод, помещающий в очередь запрос
*/
void put(NetworkRequestPrivate*);

/*!
* \brief Метод, останавливающий выполняющийся запрос
* или же убирающий его из очереди
*/
void stop(NetworkRequestPrivate*);

signals:
/*!
* \brief Прогресс отправки запроса на сервер
*/
void uploadProgress(int);

/*!
* \brief Прогресс загрузки данных с сервера
*/
void downloadProgress(int);

/*!
* \brief Данные загружены
*/
void downloadComplete(QByteArray);
void finished();

/*!
* \brief Сигнал об ошибке
*/
void error(QString);

private slots:
/*!
* \brief Слот, выполняющийся после завершения выполнения запроса
* Начинает выполнение следующего запроса в очереди
*/
void downloadComplete();

private:
/*!
* \brief Приватные конструкторы и оператор присваивания
* Для реализации паттерна Singleton
*/
NetworkQueue();
NetworkQueue(const NetworkQueue&);
NetworkQueue& operator=(const NetworkQueue&);

/*!
* \brief Извлечение запроса из очереди и его выполнение
*/
void pop();

/*!
* \brief Настройка параметров для WebLoader'а
*/
void setLoaderParams(WebLoader* _loader, NetworkRequestPrivate* _request);

/*!
* \brief Отключение сигналов WebLoader'а
* от сигналов NetworkRequestInternal
*/
void disconnectLoaderRequest(WebLoader* _loader, NetworkRequestPrivate* _request);

/*!
* \brief Очередь запросов
*/
QList<NetworkRequestPrivate*> m_queue;

/*!
* \brief Множество, содержащее запросы в очереди
* Необходим для быстрого определения, находится ли запрос в очереди
*/
QSet<NetworkRequestPrivate*> m_inQueue;

/*!
* \brief WebLoader'ы, выполняющие запрос и соответствующие им запросы
*/
QMap<WebLoader*, NetworkRequestPrivate*> m_busyLoaders;

/*!
* \brief Список свободных WebLoader'ов
*/
QList<WebLoader*> m_freeLoaders;
};

#endif // NETWORKQUEUE_H

+ 207
- 0
WebLoader/src/NetworkRequest.cpp View File

@@ -0,0 +1,207 @@
/*
* Copyright (C) 2016 Alexey Polushkin, [email protected]
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 3 of the License, or any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* Full license: http://dimkanovikov.pro/license/LGPLv3
*/

#include "NetworkRequest.h"
#include "NetworkQueue_p.h"
#include "NetworkRequestPrivate_p.h"
#include "WebLoader_p.h"
#include "WebRequest_p.h"

NetworkRequestPrivate::NetworkRequestPrivate(QObject* _parent, CustomCookieJar* _jar)
: QObject(_parent), m_cookieJar(_jar), m_loadingTimeout(20000), m_request(new WebRequest())

{

}

NetworkRequestPrivate::~NetworkRequestPrivate()
{
delete m_request;
}

void NetworkRequestPrivate::done()
{
emit finished();
}

NetworkRequest::NetworkRequest(QObject* _parent, CustomCookieJar* _jar)
: QObject(_parent), m_internal(new NetworkRequestPrivate(this, _jar))
{
//
// Соединим сигналы от internal с сигналами/слотами этого класса
//
connect(m_internal, &NetworkRequestPrivate::downloadComplete, this, &NetworkRequest::downloadComplete);
connect(m_internal, &NetworkRequestPrivate::downloadProgress,
this, &NetworkRequest::downloadProgress);
connect(m_internal, &NetworkRequestPrivate::uploadProgress,
this, &NetworkRequest::uploadProgress);
connect(m_internal, &NetworkRequestPrivate::error,
this, &NetworkRequest::slotError);
connect(m_internal, &NetworkRequestPrivate::errorDetails,
this, &NetworkRequest::slotErrorDetails);
connect(m_internal, &NetworkRequestPrivate::finished,
this, &NetworkRequest::finished);

}

NetworkRequest::~NetworkRequest()
{

}

void NetworkRequest::setCookieJar(CustomCookieJar* _cookieJar)
{
stop();
m_internal->m_cookieJar = _cookieJar;
}

CustomCookieJar* NetworkRequest::getCookieJar()
{
return m_internal->m_cookieJar;
}

void NetworkRequest::setRequestMethod(NetworkRequest::RequestMethod _method)
{
stop();
m_internal->m_method = _method;
}

NetworkRequest::RequestMethod NetworkRequest::getRequestMethod() const
{
return m_internal->m_method;
}

void NetworkRequest::setLoadingTimeout(int _loadingTimeout)
{
stop();
m_internal->m_loadingTimeout = _loadingTimeout;
}

int NetworkRequest::getLoadingTimeout() const
{
return m_internal->m_loadingTimeout;
}

void NetworkRequest::clearRequestAttributes()
{
stop();
m_internal->m_request->clearAttributes();
}

void NetworkRequest::addRequestAttribute(const QString& _name, const QVariant& _value)
{
stop();
m_internal->m_request->addAttribute(_name, _value);
}

void NetworkRequest::addRequestAttributeFile(const QString& _name, const QString& _filePath)
{
stop();
m_internal->m_request->addAttribute(_name, _filePath);
}

void NetworkRequest::setRawRequest(const QByteArray &_data)
{
stop();
m_internal->m_request->setRawRequest(_data);
}

void NetworkRequest::setRawRequest(const QByteArray &_data, const QString &_mime)
{
stop();
m_internal->m_request->setRawRequest(_data, _mime);
}

void NetworkRequest::loadAsync(const QString& _urlToLoad, const QUrl& _referer)
{
loadAsync(QUrl(_urlToLoad), _referer);
}

void NetworkRequest::loadAsync(const QUrl& _urlToLoad, const QUrl& _referer)
{
//
// Получаем инстанс очереди
//
NetworkQueue* nq = NetworkQueue::getInstance();

//
// Останавливаем в очереди запрос, связанный с данным классом (если имеется)
//
nq->stop(m_internal);

//
// Настраиваем параметры и кладем в очередь
m_internal->m_request->setUrlToLoad(_urlToLoad);
m_internal->m_request->setUrlReferer(_referer);
nq->put(m_internal);
}

QByteArray NetworkRequest::loadSync(const QString& _urlToLoad, const QUrl& _referer)
{
return loadSync(QUrl(_urlToLoad), _referer);
}

QByteArray NetworkRequest::loadSync(const QUrl& _urlToLoad, const QUrl& _referer)
{
//
// Для синхронного запроса используем QEventLoop и асинхронный запрос
//
QEventLoop loop;
connect(this, &NetworkRequest::finished, &loop, &QEventLoop::quit);
connect(m_internal, static_cast<void (NetworkRequestPrivate::*)(QByteArray, QUrl)>(&NetworkRequestPrivate::downloadComplete),
this, &NetworkRequest::downloadCompleteData);
loadAsync(_urlToLoad, _referer);
loop.exec();

return m_downloadedData;
}

QUrl NetworkRequest::url() const
{
return m_internal->m_request->urlToLoad();
}

void NetworkRequest::downloadCompleteData(const QByteArray& _data)
{
m_downloadedData = _data;
}

void NetworkRequest::slotError(const QString& _errorStr, const QUrl& _url)
{
m_lastError = _errorStr;
emit error(_errorStr, _url);
}

void NetworkRequest::slotErrorDetails(const QString& _errorStr)
{
m_lastErrorDetails = _errorStr;
}

QString NetworkRequest::lastError() const
{
return m_lastError;
}

void NetworkRequest::stop()
{
NetworkQueue* nq = NetworkQueue::getInstance();
nq->stop(m_internal);
}

QString NetworkRequest::lastErrorDetails() const
{
return m_lastErrorDetails;
}

+ 178
- 0
WebLoader/src/NetworkRequest.h View File

@@ -0,0 +1,178 @@
/*
* Copyright (C) 2016 Alexey Polushkin, [email protected]
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 3 of the License, or any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* Full license: http://dimkanovikov.pro/license/LGPLv3
*/

#ifndef NETWORKREQUEST_H
#define NETWORKREQUEST_H

#include <QEventLoop>
#include <QTimer>
#include <QUrl>

class NetworkRequestPrivate;
class WebRequest;
class CustomCookieJar;

/*!
* \brief Пользовательский класс для создания GET и POST запросов
*/

class NetworkRequest : public QObject
{
Q_OBJECT
public:

/*!
\enum Метод запроса
*/
enum RequestMethod {
Undefined, /*!< Метод не установлен */
Get,
Post
};

explicit NetworkRequest(QObject* _parent = 0, CustomCookieJar* _jar = 0);
virtual ~NetworkRequest();

/*!
* \brief Установка cookie для загрузчика
*/
void setCookieJar(CustomCookieJar* _cookieJar);

/*!
* \brief Получение cookie загрузчика
*/
CustomCookieJar* getCookieJar();

/*!
* \brief Установка метода запроса
*/
void setRequestMethod(RequestMethod _method);

/*!
* \brief Получение метода запроса
*/
RequestMethod getRequestMethod() const;

/*!
* \brief Установка таймаута загрузки
*/
void setLoadingTimeout(int _loadingTimeout);

/*!
* \brief Получение таймаута загрузки
*/
int getLoadingTimeout() const;

/*!
* \brief Очистить все старые атрибуты запроса
*/
void clearRequestAttributes();

/*!
* \brief Добавление атрибута в запрос
*/
void addRequestAttribute(const QString& _name, const QVariant& _value);

/*!
* \brief Добавление файла в запрос
*/
void addRequestAttributeFile(const QString& _name, const QString& _filePath);

void setRawRequest(const QByteArray& _data);
void setRawRequest(const QByteArray& _data, const QString& _mime);

/*!
* \brief Асинхронная загрузка запроса
*/
void loadAsync(const QString& _urlToLoad, const QUrl& _referer = QUrl());
void loadAsync(const QUrl& _urlToLoad, const QUrl& _referer = QUrl());
/*!
* \brief Синхронная загрузка запроса
*/
QByteArray loadSync(const QString& _urlToLoad, const QUrl& _referer = QUrl());
QByteArray loadSync(const QUrl& _urlToLoad, const QUrl& _referer = QUrl());

/*!
* \brief Получение загруженного URL
*/
QUrl url() const;

/*!
* \brief Получение строки с последней ошибкой
*/
QString lastError() const;
QString lastErrorDetails() const;

signals:
/*!
* \brief Прогресс отправки запроса на сервер
*/
void uploadProgress(int, QUrl);

/*!
* \brief Прогресс загрузки данных с сервера
*/
void downloadProgress(int, QUrl);

/*!
* \brief Данные загружены
*/
void downloadComplete(QByteArray, QUrl);
void finished();

/*!
* \brief Сигнал об ошибке
*/
void error(QString, QUrl);

private:

/*!
* \brief Объект, используемый в очереди запросов
*/
NetworkRequestPrivate *m_internal;

/*!
* \brief Загруженные данные в случае, если используется синхронная загрузка
*/
QByteArray m_downloadedData;

/*!
* \brief Строка, содержащая описание последней ошибки
*/
QString m_lastError;
QString m_lastErrorDetails;

/*!
* \brief Остановка выполнения запроса, связанного с текущим объектом
* и удаление запросов, ожидающих в очереди, связанных с текущим объектом
*/
void stop();

private slots:
/*!
* \brief Данные загружены. Используется при синхронной загрузке
*/
void downloadCompleteData(const QByteArray&);

/*!
* \brief Ошибка при получении данных
*/
void slotError(const QString&, const QUrl&);
void slotErrorDetails(const QString&);
};

#endif // NETWORKREQUEST_H

+ 109
- 0
WebLoader/src/NetworkRequestLoader.h View File

@@ -0,0 +1,109 @@
/*
* Copyright (C) 2016 Alexey Polushkin, [email protected]
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 3 of the License, or any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* Full license: http://dimkanovikov.pro/license/LGPLv3
*/

#ifndef NETWORKREQUESTLOADER_H
#define NETWORKREQUESTLOADER_H

#include "NetworkRequest.h"

#include <QByteArray>
#include <QUrl>

class NetworkRequestLoader {
public:
/**
* @brief Загрузить ссылку асинхронно, соединив возврат результата с заданной лямбдой
*/
template<typename Func>
static void loadAsync(const QUrl& _urlToLoad, Func _func)
{
NetworkRequest* request = new NetworkRequest;
QObject::connect(request, static_cast<void (NetworkRequest::*)(QByteArray, QUrl)>(&NetworkRequest::downloadComplete), _func);
QObject::connect(request, &NetworkRequest::finished, request, &NetworkRequest::deleteLater);
request->loadAsync(_urlToLoad);
}

/**
* @brief Загрузить ссылку асинхронно, соединив возврат результата с заданной лямбдой
*/
template<typename Func>
static void loadAsync(const QString& _urlToLoad, Func _func)
{
loadAsync(QUrl(_urlToLoad), _func);
}

/**
* @brief Загрузить список ссылок асинхронно, соединив возврат всех результатов с заданной лямбдой
*/
template<typename Container, typename Func>
static void loadAsync(const Container& _urlsList, Func _func)
{
for (auto urlToLoad : _urlsList) {
loadAsync(urlToLoad, _func);
}
}

/**
* @brief Загрузить ссылку асинхронно, соединив возврат результата с функцией класса
*/
template<typename Func>
static void loadAsync(QUrl _urlToLoad, const typename QtPrivate::FunctionPointer<Func>::Object* _reciever = 0, Func _slot = 0)
{
NetworkRequest* request = new NetworkRequest;
QObject::connect(request, static_cast<void (NetworkRequest::*)(QByteArray, QUrl)>(&NetworkRequest::downloadComplete), _reciever, _slot);
QObject::connect(request, &NetworkRequest::finished, request, &NetworkRequest::deleteLater);
request->loadAsync(_urlToLoad);
}

/**
* @brief Загрузить ссылку асинхронно, соединив возврат результата с функцией класса
*/
template<typename Func>
static void loadAsync(QString _urlToLoad, const typename QtPrivate::FunctionPointer<Func>::Object* _reciever = 0, Func _slot = 0)
{
loadAsync(QUrl(_urlToLoad), _reciever, _slot);
}

/**
* @brief Загрузить список ссылок асинхронно, соединив возврат всех результатов с функцией класса
*/
template<typename Container, typename Func>
static void loadAsync(const Container& _urlsList, const typename QtPrivate::FunctionPointer<Func>::Object* _reciever = 0, Func _slot = 0)
{
for (auto urlToLoad : _urlsList) {
loadAsync(urlToLoad, _reciever, _slot);
}
}

/**
* @brief Загрузить ссылку синхронно
*/
static QByteArray loadSync(const QUrl& _urlToLoad)
{
NetworkRequest request;
return request.loadSync(_urlToLoad);
}

/**
* @brief Загрузить ссылку синхронно
*/
static QByteArray loadSync(const QString& _urlToLoad)
{
return loadSync(QUrl(_urlToLoad));
}
};

#endif // NETWORKREQUESTLOADER_H

+ 74
- 0
WebLoader/src/NetworkRequestPrivate_p.h View File

@@ -0,0 +1,74 @@
/*
* Copyright (C) 2016 Alexey Polushkin, [email protected]
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 3 of the License, or any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* Full license: http://dimkanovikov.pro/license/LGPLv3
*/

#ifndef NETWORKREQUESTPRIVATE_H
#define NETWORKREQUESTPRIVATE_H

#include <QObject>

#include "NetworkRequest.h"

class WebLoader;

/*!
* \brief Внутренний класс для хранения в очереди запросов
* Используется только классами NetworkRequest и NetworkQueue
* Для упрощения взаимодействия данные класса являются public
*/

class NetworkRequestPrivate : public QObject
{
Q_OBJECT
public:
explicit NetworkRequestPrivate(QObject* _parent = 0, CustomCookieJar* _jar = 0);
~NetworkRequestPrivate();

QUrl m_urlToLoad;
QUrl m_referer;
CustomCookieJar* m_cookieJar;
NetworkRequest::RequestMethod m_method;
int m_loadingTimeout;
WebRequest* m_request;

void done();

signals:
/*!
* \brief Прогресс отправки запроса на сервер
*/
void uploadProgress(int, QUrl);

/*!
* \brief Прогресс загрузки данных с сервера
*/
void downloadProgress(int, QUrl);

/*!
* \brief Данные загружены
*/
void downloadComplete(QByteArray, QUrl);
void finished();

/*!
* \brief Сигнал об ошибке
*/
void error(QString, QUrl);
void errorDetails(QString, QUrl);

public slots:
};

#endif // NETWORKREQUESTPRIVATE_H

+ 340
- 0
WebLoader/src/WebLoader_p.cpp View File

@@ -0,0 +1,340 @@
/*
* Copyright (C) 2015 Dimka Novikov, [email protected]
* Copyright (C) 2016 Alexey Polushkin, [email protected]
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 3 of the License, or any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* Full license: http://dimkanovikov.pro/license/LGPLv3
*/

#include "WebLoader_p.h"
#include "WebRequest_p.h"

#include "../../customcookiejar.h"
#include <QtNetwork/QNetworkAccessManager>
#include <QtNetwork/QNetworkRequest>
#include <QtNetwork/QNetworkReply>
#include <QtNetwork/QHttpMultiPart>
#include <QtCore/QTimer>
#include <QEventLoop>
#include <QPointer>

namespace {
/**
* @brief Не все сайты передают суммарный размер загружаемой страницы,
* поэтому для отображения прогресса загрузки используется
* заранее заданное число (средний размер веб-страницы)
*/
const int POSSIBLE_RECIEVED_MAX_FILE_SIZE = 120000;

/**
* @brief Преобразовать ошибку в читаемый вид
*/
static QString networkErrorToString(QNetworkReply::NetworkError networkError) {
QString result;
switch (networkError) {
case QNetworkReply::ConnectionRefusedError: result = "the remote server refused the connection (the server is not accepting requests)"; break;
case QNetworkReply::RemoteHostClosedError: result = "the remote server closed the connection prematurely, before the entire reply was received and processed"; break;
case QNetworkReply::HostNotFoundError: result = "the remote host name was not found (invalid hostname)"; break;
case QNetworkReply::TimeoutError: result = "the connection to the remote server timed out"; break;
case QNetworkReply::OperationCanceledError: result = "the operation was canceled via calls to abort() or close() before it was finished."; break;
case QNetworkReply::SslHandshakeFailedError: result = "the SSL/TLS handshake failed and the encrypted channel could not be established. The sslErrors() signal should have been emitted."; break;
case QNetworkReply::TemporaryNetworkFailureError: result = "the connection was broken due to disconnection from the network, however the system has initiated roaming to another access point. The request should be resubmitted and will be processed as soon as the connection is re-established."; break;
case QNetworkReply::NetworkSessionFailedError: result = "the connection was broken due to disconnection from the network or failure to start the network."; break;
case QNetworkReply::BackgroundRequestNotAllowedError: result = "the background request is not currently allowed due to platform policy."; break;
case QNetworkReply::ProxyConnectionRefusedError: result = "the connection to the proxy server was refused (the proxy server is not accepting requests)"; break;
case QNetworkReply::ProxyConnectionClosedError: result = "the proxy server closed the connection prematurely, before the entire reply was received and processed"; break;
case QNetworkReply::ProxyNotFoundError: result = "the proxy host name was not found (invalid proxy hostname)"; break;
case QNetworkReply::ProxyTimeoutError: result = "the connection to the proxy timed out or the proxy did not reply in time to the request sent"; break;
case QNetworkReply::ProxyAuthenticationRequiredError: result = "the proxy requires authentication in order to honour the request but did not accept any credentials offered (if any)"; break;
case QNetworkReply::ContentAccessDenied: result = "the access to the remote content was denied (similar to HTTP error 401)"; break;
case QNetworkReply::ContentOperationNotPermittedError: result = "the operation requested on the remote content is not permitted"; break;
case QNetworkReply::ContentNotFoundError: result = "the remote content was not found at the server (similar to HTTP error 404)"; break;
case QNetworkReply::AuthenticationRequiredError: result = "the remote server requires authentication to serve the content but the credentials provided were not accepted (if any)"; break;
case QNetworkReply::ContentReSendError: result = "the request needed to be sent again, but this failed for example because the upload data could not be read a second time."; break;
case QNetworkReply::ContentConflictError: result = "the request could not be completed due to a conflict with the current state of the resource."; break;
case QNetworkReply::ContentGoneError: result = "the requested resource is no longer available at the server."; break;
case QNetworkReply::InternalServerError: result = "the server encountered an unexpected condition which prevented it from fulfilling the request."; break;
case QNetworkReply::OperationNotImplementedError: result = "the server does not support the functionality required to fulfill the request."; break;
case QNetworkReply::ServiceUnavailableError: result = "the server is unable to handle the request at this time."; break;
case QNetworkReply::ProtocolUnknownError: result = "the Network Access API cannot honor the request because the protocol is not known"; break;
case QNetworkReply::ProtocolInvalidOperationError: result = "the requested operation is invalid for this protocol"; break;
case QNetworkReply::UnknownNetworkError: result = "an unknown network-related error was detected"; break;
case QNetworkReply::UnknownProxyError: result = "an unknown proxy-related error was detected"; break;
case QNetworkReply::UnknownContentError: result = "an unknown error related to the remote content was detected"; break;
case QNetworkReply::ProtocolFailure: result = "a breakdown in protocol was detected (parsing error, invalid or unexpected responses, etc.)"; break;
case QNetworkReply::UnknownServerError: result = "an unknown error related to the server response was detected"; break;
#if QT_VERSION >= 0x050600
case QNetworkReply::TooManyRedirectsError: result = "while following redirects, the maximum limit was reached. The limit is by default set to 50 or as set by QNetworkRequest::setMaxRedirectsAllowed()."; break;
case QNetworkReply::InsecureRedirectError: result = "while following redirects, the network access API detected a redirect from a encrypted protocol (https) to an unencrypted one (http)."; break;
#endif
case QNetworkReply::NoError: result = "No error"; break;
}

return result;
}
}


WebLoader::WebLoader(QObject* _parent, CustomCookieJar* _jar) :
QThread(_parent),
m_networkManager(0),
m_cookieJar(_jar),
m_request(new WebRequest),
m_requestMethod(NetworkRequest::Undefined),
m_isNeedRedirect(true),
m_loadingTimeout(20000)
{
}

WebLoader::~WebLoader()
{
if (m_networkManager)
m_networkManager->deleteLater();//delete m_networkManager;//
}

void WebLoader::setCookieJar(CustomCookieJar* _jar)
{
if (m_cookieJar != _jar)
m_cookieJar = _jar;
}

void WebLoader::setRequestMethod(NetworkRequest::RequestMethod _method)
{
if (m_requestMethod != _method)
m_requestMethod = _method;
}

void WebLoader::setLoadingTimeout(int _msecs)
{
if (m_loadingTimeout != _msecs) {
m_loadingTimeout = _msecs;
}
}

void WebLoader::setWebRequest(WebRequest* _request) {
this->m_request = _request;
}

void WebLoader::loadAsync(const QUrl& _urlToLoad, const QUrl& _referer)
{
stop();

m_request->setUrlToLoad(_urlToLoad);
m_request->setUrlReferer (_referer);

start();
}


//*****************************************************************************
// Внутренняя реализация класса

void WebLoader::run()
{
initNetworkManager();

m_initUrl = m_request->urlToLoad();

do
{
//! Начало загрузки страницы m_request->url()
emit uploadProgress(0, m_initUrl);
emit downloadProgress(0, m_initUrl);

QPointer<QNetworkReply> reply = 0;

switch (m_requestMethod) {

default:
case NetworkRequest::Get: {
const QNetworkRequest request = this->m_request->networkRequest();
reply = m_networkManager->get(request);
break;
}

case NetworkRequest::Post: {
const QNetworkRequest networkRequest = m_request->networkRequest(true);
const QByteArray data = m_request->multiPartData();
reply = m_networkManager->post(networkRequest, data);
break;
}

} // switch

connect(reply.data(), &QNetworkReply::uploadProgress,
this, static_cast<void (WebLoader::*)(qint64, qint64)>(&WebLoader::uploadProgress));
connect(reply.data(), &QNetworkReply::downloadProgress,
this, static_cast<void (WebLoader::*)(qint64, qint64)>(&WebLoader::downloadProgress));
connect(reply.data(), static_cast<void (QNetworkReply::*)(QNetworkReply::NetworkError)>(&QNetworkReply::error),
this, &WebLoader::downloadError);
connect(reply.data(), &QNetworkReply::sslErrors, this, &WebLoader::downloadSslErrors);
connect(reply.data(), &QNetworkReply::sslErrors,
reply.data(), static_cast<void (QNetworkReply::*)()>(&QNetworkReply::ignoreSslErrors));

//
// Таймер для прерывания работы
//
QTimer timeoutTimer;
connect(&timeoutTimer, &QTimer::timeout, this, &WebLoader::quit);
connect(&timeoutTimer, &QTimer::timeout, reply.data(), &QNetworkReply::abort);
timeoutTimer.setSingleShot(true);
timeoutTimer.start(m_loadingTimeout);

//
// Входим в поток обработки событий, ожидая завершения отработки networkManager'а
//
exec();

//
// Если ответ ещё не удалён
//
if (!reply.isNull()) {
//
// ... если ответ получен, останавливаем таймер
//
if (reply->isFinished()) {
timeoutTimer.stop();
}
//
// ... а если загрузка прервалась по таймеру, освобождаем ресурсы и закрываем соединение
//
else {
m_isNeedRedirect = false;
reply->abort();
}
}

} while (m_isNeedRedirect);

emit downloadComplete(m_downloadedData, m_initUrl);
}

void WebLoader::stop()
{
if (isRunning()) {
quit();
wait(1000);
}
}

void WebLoader::uploadProgress(qint64 _uploadedBytes, qint64 _totalBytes)
{
//! отправлено [uploaded] байт из [total]
if (_totalBytes > 0)
emit uploadProgress(((float)_uploadedBytes / _totalBytes) * 100, m_initUrl);
}

void WebLoader::downloadProgress(qint64 _recievedBytes, qint64 _totalBytes)
{
//! загружено [recieved] байт из [total]
// не все сайты передают суммарный размер загружаемой страницы,
// поэтому для отображения прогресса загрузки используется
// заранее заданное число (средний размер веб-страницы)
if (_totalBytes < 0)
_totalBytes = POSSIBLE_RECIEVED_MAX_FILE_SIZE;
emit downloadProgress(((float)_recievedBytes / _totalBytes) * 100, m_initUrl);
}

void WebLoader::downloadComplete(QNetworkReply* _reply)
{
//! Завершена загрузка страницы [m_request->url()]

// требуется ли редирект?
if (!_reply->header(QNetworkRequest::LocationHeader).isNull()) {
//! Осуществляется редирект по ссылке [redirectUrl]
// Referer'ом становится ссылка по хоторой был осуществлен запрос
QUrl refererUrl = m_request->urlToLoad();
m_request->setUrlReferer(refererUrl);
// Получаем ссылку для загрузки из заголовка ответа [Loacation]
QUrl redirectUrl = _reply->header(QNetworkRequest::LocationHeader).toUrl();
m_request->setUrlToLoad(redirectUrl);
setRequestMethod(NetworkRequest::Get); // Редирект всегда методом Get
m_isNeedRedirect = true;
} else {
//! Загружены данные [reply->bytesAvailable()]
qint64 downloadedDataSize = _reply->bytesAvailable();
QByteArray downloadedData = _reply->read(downloadedDataSize);
m_downloadedData = downloadedData;
_reply->deleteLater();
m_isNeedRedirect = false;
}

if (!isRunning()) {
wait(1000);
}
quit(); // прерываем цикл обработки событий потока (возвращаемся в run())
}

void WebLoader::downloadError(QNetworkReply::NetworkError _networkError)
{
switch (_networkError) {

case QNetworkReply::NoError:
m_lastError.clear();
m_lastErrorDetails.clear();
break;
default:
m_lastError =
tr("%1")
.arg(::networkErrorToString(_networkError));
emit error(m_lastError, m_initUrl);
break;

}
}

void WebLoader::downloadSslErrors(const QList<QSslError>& _errors)
{
QString fullError;
foreach (const QSslError& error, _errors) {
if (!fullError.isEmpty()) {
fullError.append("\n");
}
fullError.append(error.errorString());
}

m_lastErrorDetails = fullError;
emit errorDetails(m_lastErrorDetails, m_initUrl);
}


//*****************************************************************************
// Методы доступа к данным класса, а так же вспомогательные
// методы для работы с данными класса

void WebLoader::initNetworkManager()
{
//
// Создаём загрузчика, если нужно
//
if (m_networkManager == 0) {
m_networkManager = new QNetworkAccessManager;
}

//
// Настраиваем куки
//
if (m_cookieJar != 0) {
m_networkManager->setCookieJar(m_cookieJar);
m_cookieJar->setParent(0);
}

//
// Оключаем от предыдущих соединений
//
m_networkManager->disconnect();
//
// Настраиваем новое соединение
//
connect(m_networkManager, &QNetworkAccessManager::finished,
this, static_cast<void (WebLoader::*)(QNetworkReply*)>(&WebLoader::downloadComplete));
}

+ 164
- 0
WebLoader/src/WebLoader_p.h View File

@@ -0,0 +1,164 @@
/*
* Copyright (C) 2015 Dimka Novikov, [email protected]
* Copyright (C) 2016 Alexey Polushkin, [email protected]
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 3 of the License, or any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* Full license: http://dimkanovikov.pro/license/LGPLv3
*/

#ifndef WEBLOADER_H
#define WEBLOADER_H

#include "NetworkRequest.h"
#include "../../customcookiejar.h"

#include <QtCore/QThread>
#include <QtCore/QUrl>
#include <QtNetwork/QNetworkReply>

class QNetworkAccessManager;
class CustomCookieJar;
class WebRequest;

/*!
\class WebLoader

\brief Класс для работы с http-протоколом
*/
class WebLoader : public QThread
{
Q_OBJECT

public:

public:
explicit WebLoader(QObject* _parent = 0, CustomCookieJar* _jar = 0);
virtual ~WebLoader();

/*!
\brief Установка куков для сессии загрузчика
*/
void setCookieJar(CustomCookieJar* _cookieJar);

/*!
\brief Установка метода запроса
*/
void setRequestMethod(NetworkRequest::RequestMethod _method);

/**
* @brief Установить таймаут загрузки
*/
void setLoadingTimeout(int _msecs);

/*!
* \brief Установка WebRequest
*/
void setWebRequest(WebRequest* _request);

/*!
\brief Отправка запроса (асинхронное выполнение)
*/
void loadAsync(const QUrl& _urlToLoad, const QUrl& _referer = QUrl());

/**
* @brief Остановить выполнение
*/
void stop();

signals:
/*!
\brief Прогресс отправки запроса на сервер
*/
void uploadProgress(int, QUrl);

/*!
\brief Прогресс загрузки данных с сервера
*/
void downloadProgress(int, QUrl);

/*!
\brief Данные загружены
*/
void downloadComplete(QByteArray, QUrl);

/*!
\brief Сигнал об ошибке
*/
void error(QString, QUrl);
void errorDetails(QString, QUrl);


//*****************************************************************************
// Внутренняя реализация класса

private:
/*!
\brief Работа потока
*/
void run();


private slots:
/*!
\brief Прогресс отправки запроса на сервер
* uploadedBytes - отправлено байт, totalBytes - байт к отправке
*/
void uploadProgress(qint64 _uploadedBytes, qint64 _totalBytes);

/*!
\brief Прогресс загрузки данных с сервера
* recievedBytes загружено байт, totalBytes - байт к отправке
*/
void downloadProgress(qint64 _recievedBytes, qint64 _totalBytes);

/*!
\brief Окончание загрузки страницы
*/
void downloadComplete(QNetworkReply* _reply);

/*!
\brief Ошибка при загрузки страницы
*/
void downloadError(QNetworkReply::NetworkError _networkError);

/*!
* \brief Ошибки при защищённом подключении
*/
void downloadSslErrors(const QList<QSslError>& _errors);


//*****************************************************************************
// Методы доступа к данным класса, а так же вспомогательные
// методы для работы с данными класса
private:
void initNetworkManager();

// Данные класса
private:
QNetworkAccessManager* m_networkManager;
CustomCookieJar* m_cookieJar;
WebRequest* m_request;
NetworkRequest::RequestMethod m_requestMethod;
bool m_isNeedRedirect;
QUrl m_initUrl;

/**
* @brief Таймаут загрузки ссылки
*/
int m_loadingTimeout;

QByteArray m_downloadedData;
QString m_lastError;
QString m_lastErrorDetails;
};

#endif // WEBLOADER_H

+ 212
- 0
WebLoader/src/WebRequest_p.cpp View File

@@ -0,0 +1,212 @@
/*
* Copyright (C) 2015 Dimka Novikov, [email protected]
* Copyright (C) 2016 Alexey Polushkin, [email protected]
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 3 of the License, or any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* Full license: http://dimkanovikov.pro/license/LGPLv3
*/

#include "WebRequest_p.h"
#include "HttpMultiPart_p.h"


#include <QFile>
#include <QStringList>
#include <QSslConfiguration>
#include <QMimeDatabase>

//! Заголовки запроса
const QByteArray USER_AGENT_HEADER = "User-Agent";
const QByteArray REFERER_HEADER = "Referer";
//! Параметры запроса
// UserAgent запроса
const QByteArray USER_AGENT = "Web Robot v.0.12.1";
// Boundary для разделения атрибутов запроса, при использовании multi-part form data
const QString BOUNDARY = "---------------------------7d935033608e2";
// ContentType запроса
const QString CONTENT_TYPE_DEFAULT = "application/x-www-form-urlencoded";
const QString CONTENT_TYPE = "multipart/form-data; boundary=" + BOUNDARY;


WebRequest::WebRequest()
{

}

WebRequest::~WebRequest()
{

}

QUrl WebRequest::urlToLoad() const
{
return m_urlToLoad;
}

void WebRequest::setUrlToLoad(const QUrl& _url)
{
if (urlToLoad() != _url)
m_urlToLoad = _url;
}

QUrl WebRequest::urlReferer() const
{
return m_urlReferer;
}

void WebRequest::setUrlReferer(const QUrl& _url)
{
if (urlReferer() != _url)
m_urlReferer = _url;
}

void WebRequest::clearAttributes()
{
m_attributes.clear();
m_attributeFiles.clear();
m_rawData.clear();
}

void WebRequest::addAttribute(const QString& _name, const QVariant& _value)
{
if(m_usedRaw && !m_rawData.isEmpty()) {
qWarning() << "You are trying to mix methods. Raw data will be cleaned";
m_rawData.clear();
}
m_usedRaw = false;

QPair< QString, QVariant > attribute;
attribute.first = _name;
attribute.second = _value;
addAttribute(attribute);
}

void WebRequest::addAttributeFile(const QString& _name, const QString& _filePath)
{
if(m_usedRaw && !m_rawData.isEmpty()) {
qWarning() << "You are trying to mix methods. Raw data will be cleaned";
m_rawData.clear();
}
m_usedRaw = false;

QPair< QString, QString > attributeFile;
attributeFile.first = _name;
attributeFile.second = _filePath;
addAttributeFile(attributeFile);
}

void WebRequest::setRawRequest(const QByteArray &_data)
{
setRawRequest(_data, QMimeDatabase().mimeTypeForData(_data).name());
}

void WebRequest::setRawRequest(const QByteArray &_data, const QString &_mime)
{
if(!m_usedRaw && (!m_attributes.isEmpty() || !m_attributeFiles.isEmpty())) {
qWarning() << "You are trying to mix methods. Attributes will be cleaned";
m_attributes.clear();
m_attributeFiles.clear();
}
m_usedRaw = true;
m_rawData = _data;
m_mimeRawData = _mime;
}

QNetworkRequest WebRequest::networkRequest(bool _addContentHeaders)
{
QNetworkRequest request(urlToLoad());
// Установка заголовков запроса
// User-Agent
request.setRawHeader(USER_AGENT_HEADER, USER_AGENT);
// Referer
if (!urlReferer().isEmpty())
request.setRawHeader(REFERER_HEADER, urlReferer().toString().toUtf8().data());