Updated the README and the mixing to provide better asserts.

.gitea/workflows/publish.yaml

@@ -0,0 +1,51 @@
+name: StarFields Django Rest Framework Generics
+
+on: [push]
+
+env:
+  GITHUB_WORKFLOW_REF:
+  TWINE_USERNAME: ${{ secrets.GIT_PYPI_USERNAME }}
+  TWINE_PASSWORD: ${{ secrets.GIT_PYPI_PASSWORD }}
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    # Bizzarre bug that needed the below according to https://forum.gitea.com/t/gitea-actions-with-python/7605/3
+    container: catthehacker/ubuntu:act-latest
+
+    steps:
+      - name: Checkout code
+        run: |
+            git clone https://${{secrets.GIT_USERNAME}}:${{secrets.GIT_PASSWORD_URLENCODED}}@git.starfieldsweb.com/StarFields/starfields-drf-generics ./
+
+      - name: Set up Python
+        # This is the version of the action for setting up Python, not the Python version.
+        uses: https://github.com/actions/setup-python@v5
+        with:
+          # Semantic version range syntax or exact version of a Python version
+          python-version: '3.12.3'
+
+      - name: Display Python version
+        run: python -c "import sys; print(sys.version)"
+
+      # TODO testing
+      # check https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python#packaging-workflow-data-as-artifacts
+
+      - name: Build package
+        run: pip install build && python -m build
+
+      - name: Publish package to Gitea PyPI
+        run: pip install twine && pip install --upgrade pkginfo && python -m twine upload --repository-url https://git.starfieldsweb.com/api/packages/StarFields/pypi ./dist/*
+
+      # - name: Publish package to Gitea PyPI
+      #   continue-on-error: false
+      #   uses: pypa/gh-action-pypi-publish@release/v1
+      #   with:
+      #       user: ${{ secrets.GIT_PYPI_USERNAME }}
+      #       password: ${{ secrets.GIT_PYPI_PASSWORD }}
+      #       repository-url: https://git.starfieldsweb.com/api/packages/StarFields/pypi
+      #       print-hash: true
+      #       verbose: true
+
+# TODO make a release section that creates a gitea release

.gitignore

@@ -0,0 +1,5 @@
+# Ignore all the python cache files
+**/__pycache__/**/*
+
+# Ignore all the dist files
+**/dist/**/*

README.md

@@ -2,5 +2,84 @@ This repository holds the django library that StarFields uses for the django-res
 
 # Differences with the DRF generic views
 
-It changes the generic lifecycles of all the CRUD operations to fit within them automated caching functionality. Caching and deleting cache keys is handled by the library in a way that the cache keys have no duplicates. The generic views offered include single item CRUD and list-based CRUD.
+The generic views of DRF use the serializers in such a way that the model serializers directly integrate the functionality of a specific model (db table) with CRUD. As a result simple operations such as deleting a list of n table rows ends up using n queries. This serves automated CRUD well but control and performance suffer, in particular changing the view and serializer methods to use a single query in order to perform a write operation becomes a very non-uniform experience between request methods. This library changes the DRF generic views to be more uniform in the way they use the serializers, in particular a single query is made for GET requests with elaborate filtering capabilities and calls to serializer .create(), .update() and .destroy() methods for write methods.
+
+In particular:
+ - Single Create operations create the model instance as expected
+ - Single Retrive, Update and Destroy work with the .get_object() callable to find and manipulate the instance
+ - List Retrieve operations work through elaborate filters to get results starting with the .get_queryset() callable.
+ - List Create, Update and Destroy need to implement ListSerializer .create(), .update() and .destroy() methods for bulk operations. None of those methods use the .get_queryset() callable.
+
+It is easy to notice that the single item apis are useful but limited. List apis allow you to perform much more flexible operations and organize frontends better as well, allowing for example out of step syncing (such as shopping carts). As a result it is recommended to use strictly list based generic views for all apis that are supposed to be flexibly used and restrict yourself to using single item generic views to apis whose access you want to partially restrict to the outside world from scanning (such as users) or are inherently simple.
+
+The generics are also enhanced to fit within them automated caching functionality. Caching and deleting cache keys is handled by the library in a way that the cache keys have no duplicates. The generic views offered include single item CRUD and list-based CRUD.
+
 To manage automated caching this the library replaces (and appends to) the DRF filters. These filters need a get_unique_dict method in order to avoid the duplicate cache keys problem.
+
+# Usage
+### Making views in views.py:
+```python
+from starfields_drf_generics import generics
+from starfields_drf_generics import filters as libfilters
+
+class CategoriesView(generics.CachedListRetrieveAPIView):
+    """
+    This view lists all the categories. Usually this API is called on the shop initialization and the result is saved on the front-end for use shop-wide.
+    """
+    cache_prefix = "shop.products.categories"
+    cache_vary_on_user = False
+    cache_timeout_mins = ShopSettings.get_solo().cache_timeout
+    queryset = Category.objects.all()
+    serializer_class = CategorySerializer
+    paged = False
+    filter_backends = []
+    logger = logger
+    cache = cache
+
+class SearchView(generics.CachedListRetrieveAPIView):
+    """
+    This view lists the gallery pictures with extensive searching and filtering features. You can use this API to get the latest pictures, perform picture searches among others.
+    """
+    cache_prefix = "gallery.search"
+    cache_vary_on_user = False
+    cache_timeout_mins = GallerySettings.get_solo().cache_timeout
+    queryset = Picture.objects.filter(published=True)
+    serializer_class = PictureSerializer
+    ordering_fields = ('similarity','date_added','updated')
+    category_class = Category
+    search_fields = ['name','slug','tag__name','tag__slug']
+    paged = True
+    default_page_size = 20
+    filter_backends = (libfilters.CategoryFilter,
+                       libfilters.TrigramSearchFilter,
+                       libfilters.OrderingFilter,
+                       )
+    logger = logger
+    cache = cache
+```
+
+### New class attributes that are used
+| Attribute             | Description                                                                    |
+| ---------             | -----------                                                                    |
+| cache_prefix          | Defines the prefix that the module will use when saving values in the cache    |
+| cache_vary_on_user    | Defines whether keys saved in the cache are different for each user, in which case extra user information will be added to the cache prefix |
+| cache_timeout_mins    | The cache key timeout                                                          |
+| filter_backends       | The filters that you want the view to have, each can be configured with view class attributes |
+| ordering_fields       | If you use the OrderingFilter you must indicate what fields the user can order by, the first element is used as the default order |
+| search_fields         | If you use the TrigramSearchFilter you must indicate the fields to search through |
+| paged                 | Your generic view can have a pager for the user to choose pages or it can be a full listing |
+| default_page_size     | The default size of the pages if a user has not indicated a page size |
+| logger                | You should register a logger in order to get error feedback in your deployments |
+| cache                 | The main feature of this module is automated and organized caching, you should register your cache here |
+
+
+### Extras
+The source code is similar to the django-rest-framework's generic classes and related objects, it should be eminently readable. As with the rest framework's generics multiple inheritance is used through mixing to organize the behavior of each class in a standalone setting.
+
+# Duplicate Cache Keys Problem
+The problem arises in the way cache keys are created, the naive method is to just use the information from the url of the request and just save it to the cache. This creates a problem in that a request such as https://example.com/api/v1/pictures/search?search=mypic&category=mycat and a request https://example.com/api/v1/pictures/search?category=mycat&search=mypic contain the same information in their cache values. So the order of each filter or the order within the filters (such as the facet filter I made for e-commerce APIs) affects the caching behavior and creates more work for our APIs.
+
+The way this module fixes the duplicate cache keys problem is by systematically ordering the filters through each filter's get_unique_dict method that are called in cache_mixins.py and then running the sorted_params_string utility function in the resulting dict.
+
+# Why this instead of webserver caching
+Using web server caching, in particular API microcaching, eg [Benefits of Microcaching](https://www.nginx.com/blog/benefits-of-microcaching-nginx/), is recommended to be used along with this library. This way the microcaching at the web server level manages the bulk of the caching while this cache that sits further back manages more flexible caching. This permits among others, runtime cache timeout configuration, handling of the duplicate cache keys problem above, more cache redundancy and more flexible and complicated network topologies for your servers.

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "starfields-drf-generics"
-version = "0.1.0"
+version = "0.4.1"
 authors = [
   { name="Anastasios Svolis", email="support@starfields.gr" },
 ]
@@ -18,5 +18,14 @@ classifiers = [
 ]
 
 [project.urls]
-"Homepage" = "https://git.vickys-corner.xyz/ace/starfields-drf-generics"
+"Homepage" = "https://git.starfieldsweb.com/StarFields/starfields-drf-generics"
 
+[tool.setuptools.packages.find]
+where = ["starfields_drf_generics"]
+
+[tool.setuptools.package-data]
+"templates.filters" = ["*.html"]
+
+[tool.hatch.build.targets.wheel]
+include = ["*.py", "*.html"]
+exclude = ["test*"]

starfields_drf_generics/cache_mixins.py

@@ -1,32 +1,41 @@
 from starfields_drf_generics.utils import sorted_params_string
 
-# TODO classes below that involve create, update, destroy don't delete the caches properly, they need a regex cache delete
+# TODO classes below that involve create, update, destroy don't delete the
+# caches properly, they need a regex cache delete
 # TODO put more reasonable asserts and feedback
 
+
 # Mixin classes that provide cache functionalities
 class CacheUniqueUrl:
     def get_cache_unique_url(self, request):
-        """ Create the query to be cached in a unique way to avoid duplicates. """
+        """
+        Create the query to be cached in a unique way to avoid duplicates.
+        """
         if not hasattr(self, 'filters_string'):
             # Only assign the attribute if it's not already assigned
             filters = {}
             if self.extra_filters_dict:
                 filters.update(self.extra_filters_dict)
-                # Check if the url parameters have any of the keys of the extra filters and if so assign them
+                # Check if the url parameters have any of the keys of the extra
+                # filters and if so assign them
                 for key in self.extra_filters_dict:
                     if key in self.request.query_params.keys():
-                        filters[key] = self.request.query_params[key].replace(' ','').split(',')
+                        filters[key] = self.request.query_params[key].replace(
+                            ' ', '').split(',')
                     # Check if they're resolved in the urlconf as well
                     if key in self.kwargs.keys():
                         filters[key] = [self.kwargs[key]]
 
             if hasattr(self, 'paged'):
                 if self.paged:
-                    filters.update({'limit': [self.default_page_size], 'offset': [0]})
+                    filters.update({'limit': [self.default_page_size],
+                                    'offset': [0]})
                     if 'limit' in self.request.query_params.keys():
-                        filters.update({'limit': [self.request.query_params['limit']]})
+                        filters.update({
+                            'limit': [self.request.query_params['limit']]})
                     if 'offset' in self.request.query_params.keys():
-                        filters.update({'offset': [self.request.query_params['offset']]})
+                        filters.update({
+                            'offset': [self.request.query_params['offset']]})
             for backend in list(self.filter_backends):
                 filters.update(backend().get_filters_dict(request, self))
             self.filters_string = sorted_params_string(filters)
@@ -50,11 +59,15 @@ class CacheGetMixin(CacheUniqueUrl):
         # Attempt to get the response from the cache for the whole request
         try:
             if self.cache_vary_on_user:
-                cache_attempt = self.cache.get(f"{self.cache_prefix}.{request.user}.{self.filters_string}")
+                cache_attempt = self.cache.get(
+                    f"{self.cache_prefix}.{request.user}.{self.filters_string}"
+                    )
             else:
-                cache_attempt = self.cache.get(f"{self.cache_prefix}.{self.filters_string}")
-        except:
-            self.logger.info(f"Cache get attempt for {self.__class__.__name__} failed.")
+                cache_attempt = self.cache.get(
+                    f"{self.cache_prefix}.{self.filters_string}")
+        except Exception:
+            self.logger.info(f"Cache get attempt for {self.__class__.__name__}"
+                             " failed.")
             cache_attempt = None
 
         if cache_attempt:
@@ -79,15 +92,18 @@ class CacheSetMixin(CacheUniqueUrl):
                 # Writes the response to the cache
                 try:
                     if self.cache_vary_on_user:
-                        self.cache.set(key=f"{self.cache_prefix}.{request.user}.{self.filters_string}",
+                        self.cache.set(key=f"{self.cache_prefix}."
+                                       f"{request.user}.{self.filters_string}",
                                        value=response.data,
                                        timeout=60*self.cache_timeout_mins)
                     else:
-                        self.cache.set(key=f"{self.cache_prefix}.{self.filters_string}",
+                        self.cache.set(key=f"{self.cache_prefix}"
+                                       f".{self.filters_string}",
                                        value=response.data,
                                        timeout=60*self.cache_timeout_mins)
-                except:
-                    self.logger.exception(f"Cache set attempt for {self.__class__.__name__} failed.")
+                except Exception:
+                    self.logger.exception("Cache set attempt for "
+                                          f"{self.__class__.__name__} failed.")
             return caching_function
 
         # Register the post rendering hook to the response
@@ -109,14 +125,18 @@ class CacheDeleteMixin(CacheUniqueUrl):
             self.get_cache_unique_url(request)
 
             assert self.cache_prefix is not None, (
-                f"{self.__class__.__name__} should include a `cache_prefix` attribute"
+                f"{self.__class__.__name__} should include a `cache_prefix`"
+                "attribute"
             )
 
             # Delete the cache since a new entry has been created
             try:
                 if self.cache_vary_on_user:
-                    self.cache.delete(f"{self.cache_prefix}.{request.user}.{self.filters_string}")
+                    self.cache.delete(f"{self.cache_prefix}.{request.user}"
+                                      f".{self.filters_string}")
                 else:
-                    self.cache.delete(f"{self.cache_prefix}.{self.filters_string}")
-            except:
-                self.logger.exception(f"Cache delete attempt for {self.__class__.__name__} failed.")
+                    self.cache.delete(f"{self.cache_prefix}"
+                                      f".{self.filters_string}")
+            except Exception:
+                self.logger.exception("Cache delete attempt for "
+                                      f"{self.__class__.__name__} failed.")

starfields_drf_generics/generics.py

@@ -1,15 +1,7 @@
 """
 Generic views that provide commonly needed behaviour.
 """
-from django.core.exceptions import ValidationError
-from django.db.models.query import QuerySet
-from django.http import Http404
-from django.shortcuts import get_object_or_404 as _get_object_or_404
-
-from rest_framework import views
 from rest_framework.generics import GenericAPIView
-from rest_framework.settings import api_settings
-
 from starfields_drf_generics import mixins
 
 
@@ -18,26 +10,32 @@ from starfields_drf_generics import mixins
 
 # Single item CRUD
 
-class CachedCreateAPIView(mixins.CachedCreateModelMixin,GenericAPIView):
+class CachedCreateAPIView(mixins.CachedCreateModelMixin,
+                          GenericAPIView):
     """
     Concrete view for creating a model instance.
     """
+
     def post(self, request, *args, **kwargs):
         return self.create(request, *args, **kwargs)
 
 
-class CachedRetrieveAPIView(mixins.CachedRetrieveModelMixin,GenericAPIView):
+class CachedRetrieveAPIView(mixins.CachedRetrieveModelMixin,
+                            GenericAPIView):
     """
     Concrete view for retrieving a model instance.
     """
+
     def get(self, request, *args, **kwargs):
         return self.retrieve(request, *args, **kwargs)
 
 
-class CachedUpdateAPIView(mixins.CachedUpdateModelMixin,GenericAPIView):
+class CachedUpdateAPIView(mixins.CachedUpdateModelMixin,
+                          GenericAPIView):
     """
     Concrete view for updating a model instance.
     """
+
     def put(self, request, *args, **kwargs):
         return self.update(request, *args, **kwargs)
 
@@ -45,18 +43,23 @@ class CachedUpdateAPIView(mixins.CachedUpdateModelMixin,GenericAPIView):
         return self.partial_update(request, *args, **kwargs)
 
 
-class CachedDestroyAPIView(mixins.CachedDestroyModelMixin,GenericAPIView):
+class CachedDestroyAPIView(mixins.CachedDestroyModelMixin,
+                           GenericAPIView):
     """
     Concrete view for deleting a model instance.
     """
+
     def delete(self, request, *args, **kwargs):
         return self.destroy(request, *args, **kwargs)
 
 
-class CachedRetrieveUpdateAPIView(mixins.CachedRetrieveModelMixin,mixins.CachedUpdateModelMixin,GenericAPIView):
+class CachedRetrieveUpdateAPIView(mixins.CachedRetrieveModelMixin,
+                                  mixins.CachedUpdateModelMixin,
+                                  GenericAPIView):
     """
     Concrete view for retrieving, updating a model instance.
     """
+
     def get(self, request, *args, **kwargs):
         return self.retrieve(request, *args, **kwargs)
 
@@ -67,10 +70,13 @@ class CachedRetrieveUpdateAPIView(mixins.CachedRetrieveModelMixin,mixins.CachedU
         return self.partial_update(request, *args, **kwargs)
 
 
-class CachedRetrieveDestroyAPIView(mixins.CachedRetrieveModelMixin,mixins.CachedDestroyModelMixin,GenericAPIView):
+class CachedRetrieveDestroyAPIView(mixins.CachedRetrieveModelMixin,
+                                   mixins.CachedDestroyModelMixin,
+                                   GenericAPIView):
     """
     Concrete view for retrieving or deleting a model instance.
     """
+
     def get(self, request, *args, **kwargs):
         return self.retrieve(request, *args, **kwargs)
 
@@ -78,10 +84,14 @@ class CachedRetrieveDestroyAPIView(mixins.CachedRetrieveModelMixin,mixins.Cached
         return self.destroy(request, *args, **kwargs)
 
 
-class CachedRetrieveUpdateDestroyAPIView(mixins.CachedRetrieveModelMixin,mixins.CachedUpdateModelMixin,mixins.CachedDestroyModelMixin,GenericAPIView):
+class CachedRetrieveUpdateDestroyAPIView(mixins.CachedRetrieveModelMixin,
+                                         mixins.CachedUpdateModelMixin,
+                                         mixins.CachedDestroyModelMixin,
+                                         GenericAPIView):
     """
     Concrete view for retrieving, updating or deleting a model instance.
     """
+
     def get(self, request, *args, **kwargs):
         return self.retrieve(request, *args, **kwargs)
 
@@ -95,10 +105,16 @@ class CachedRetrieveUpdateDestroyAPIView(mixins.CachedRetrieveModelMixin,mixins.
         return self.destroy(request, *args, **kwargs)
 
 
-class CachedCreateRetrieveUpdateDestroyAPIView(mixins.CachedCreateModelMixin,mixins.CachedRetrieveModelMixin,mixins.CachedUpdateModelMixin,mixins.CachedDestroyModelMixin,GenericAPIView):
+class CachedCreateRetrieveUpdateDestroyAPIView(mixins.CachedCreateModelMixin,
+                                               mixins.CachedRetrieveModelMixin,
+                                               mixins.CachedUpdateModelMixin,
+                                               mixins.CachedDestroyModelMixin,
+                                               GenericAPIView):
     """
-    Concrete view for creating, retrieving, updating or deleting a model instance.
+    Concrete view for creating, retrieving, updating or deleting a model
+    instance.
     """
+
     def get(self, request, *args, **kwargs):
         return self.retrieve(request, *args, **kwargs)
 
@@ -117,26 +133,32 @@ class CachedCreateRetrieveUpdateDestroyAPIView(mixins.CachedCreateModelMixin,mix
 
 # List based CRUD
 
-class CachedListRetrieveAPIView(mixins.CachedListRetrieveModelMixin,GenericAPIView):
+class CachedListRetrieveAPIView(mixins.CachedListRetrieveModelMixin,
+                                GenericAPIView):
     """
     Concrete view for listing a queryset.
     """
+
     def get(self, request, *args, **kwargs):
         return self.list(request, *args, **kwargs)
 
 
-class CachedListCreateAPIView(mixins.CachedListCreateModelMixin,GenericAPIView):
+class CachedListCreateAPIView(mixins.CachedListCreateModelMixin,
+                              GenericAPIView):
     """
     Concrete view for creating multiple instances.
     """
+
     def post(self, request, *args, **kwargs):
         return self.list_create(request, *args, **kwargs)
 
 
-class CachedListUpdateAPIView(mixins.CachedListUpdateModelMixin,GenericAPIView):
+class CachedListUpdateAPIView(mixins.CachedListUpdateModelMixin,
+                              GenericAPIView):
     """
     Concrete view for updating multiple instances.
     """
+
     def put(self, request, *args, **kwargs):
         return self.list_update(request, *args, **kwargs)
 
@@ -144,18 +166,23 @@ class CachedListUpdateAPIView(mixins.CachedListUpdateModelMixin,GenericAPIView):
         return self.list_partial_update(request, *args, **kwargs)
 
 
-class CachedListDestroyAPIView(mixins.CachedListDestroyModelMixin,GenericAPIView):
+class CachedListDestroyAPIView(mixins.CachedListDestroyModelMixin,
+                               GenericAPIView):
     """
     Concrete view for deleting multiple instances.
     """
+
     def delete(self, request, *args, **kwargs):
         return self.list_destroy(request, *args, **kwargs)
 
 
-class CachedListRetrieveCreateAPIView(mixins.CachedListRetrieveModelMixin,mixins.CachedListCreateModelMixin,GenericAPIView):
+class CachedListRetrieveCreateAPIView(mixins.CachedListRetrieveModelMixin,
+                                      mixins.CachedListCreateModelMixin,
+                                      GenericAPIView):
     """
     Concrete view for listing a queryset or creating a model instance.
     """
+
     def get(self, request, *args, **kwargs):
         return self.list(request, *args, **kwargs)
 
@@ -163,10 +190,15 @@ class CachedListRetrieveCreateAPIView(mixins.CachedListRetrieveModelMixin,mixins
         return self.create(request, *args, **kwargs)
 
 
-class CachedListCreateRetrieveDestroyAPIView(mixins.CachedListCreateModelMixin,mixins.CachedListRetrieveModelMixin,mixins.CachedListDestroyModelMixin,GenericAPIView):
+class CachedListCreateRetrieveDestroyAPIView(
+      mixins.CachedListCreateModelMixin,
+      mixins.CachedListRetrieveModelMixin,
+      mixins.CachedListDestroyModelMixin,
+      GenericAPIView):
     """
     Concrete view for creating, retrieving or deleting a model instance.
     """
+
     def get(self, request, *args, **kwargs):
         return self.list(request, *args, **kwargs)
 
@@ -177,10 +209,16 @@ class CachedListCreateRetrieveDestroyAPIView(mixins.CachedListCreateModelMixin,m
         return self.list_destroy(request, *args, **kwargs)
 
 
-class CachedListCreateRetrieveUpdateAPIView(mixins.CachedListCreateModelMixin,mixins.CachedListRetrieveModelMixin,mixins.CachedListUpdateModelMixin,GenericAPIView):
+class CachedListCreateRetrieveUpdateAPIView(
+      mixins.CachedListCreateModelMixin,
+      mixins.CachedListRetrieveModelMixin,
+      mixins.CachedListUpdateModelMixin,
+      GenericAPIView):
     """
-    Concrete view for creating, retrieving, updating or deleting a model instance.
+    Concrete view for creating, retrieving, updating or deleting a model
+    instance.
     """
+
     def get(self, request, *args, **kwargs):
         return self.list(request, *args, **kwargs)
 
@@ -194,10 +232,17 @@ class CachedListCreateRetrieveUpdateAPIView(mixins.CachedListCreateModelMixin,mi
         return self.list_partial_update(request, *args, **kwargs)
 
 
-class CachedListCreateRetrieveUpdateDestroyAPIView(mixins.CachedListCreateModelMixin,mixins.CachedListRetrieveModelMixin,mixins.CachedListUpdateModelMixin,mixins.CachedListDestroyModelMixin,GenericAPIView):
+class CachedListCreateRetrieveUpdateDestroyAPIView(
+      mixins.CachedListCreateModelMixin,
+      mixins.CachedListRetrieveModelMixin,
+      mixins.CachedListUpdateModelMixin,
+      mixins.CachedListDestroyModelMixin,
+      GenericAPIView):
     """
-    Concrete view for creating, retrieving, updating or deleting a model instance.
+    Concrete view for creating, retrieving, updating or deleting a model
+    instance.
     """
+
     def get(self, request, *args, **kwargs):
         return self.list(request, *args, **kwargs)
 

starfields_drf_generics/libfilters/category.py

@@ -0,0 +1,126 @@
+from rest_framework.filters import BaseFilterBackend
+from django.template import loader
+from django.db.models import Q
+
+
+class CategoryFilter(BaseFilterBackend):
+    """
+    This filter assigns the view.category object for later use, in particular
+    for filters that depend on this one.
+    """
+    template = 'filters/categories.html'
+    category_field = 'category'
+
+    def get_category_class(self, view, request):
+        return getattr(view, 'category_class', None)
+
+    def assign_view_category(self, request, view):
+        if not hasattr(view, self.category_field):
+            if self.category_field in request.query_params.keys():
+                try:
+                    category_slug = request.query_params.get(
+                        self.category_field).strip()
+                    category = view.category_class.objects.get(
+                        slug=category_slug)
+                    # Append the category object in the view for later use
+                    view.category = category
+                except Exception:
+                    view.category = None
+            else:
+                view.category = None
+
+    def get_filters_dict(self, request, view):
+        """
+        Custom method that returns the filters exclusive to this filter in a
+        dict. For caching purposes. Queries the database for the current
+        category and saves it in view.category for internal use and later
+        filters.
+        """
+        if hasattr(view, 'category_field'):
+            self.category_field = self.get_category_field(view, request)
+
+        category_class = self.get_category_class(view, request)
+
+        assert category_class is not None, (
+            f"{view.__class__.__name__} should include a `category_class`"
+            "attribute"
+        )
+
+        # Create the filters dictionary and find the present category instance
+        self.assign_view_category(request, view)
+
+        filters_dict = {}
+        if view.category:
+            filters_dict[self.category_field] = [view.category.slug]
+        else:
+            filters_dict[self.category_field] = []
+
+        return filters_dict
+
+    def filter_queryset(self, request, queryset, view):
+        self.assign_view_category(request, view)
+
+        # Create the queryset
+        if view.category:
+            kwquery_1 = {}
+            kwquery_1[self.category_field+'__id'] = view.category.id
+            if view.category.tn_descendants_pks:
+                kwquery_2 = {}
+                key = self.category_field+'__id__in'
+                kwquery_2[key] = view.category.tn_descendants_pks.split(',')
+
+                queryset = queryset.filter(Q(**kwquery_1) | Q(**kwquery_2))
+            else:
+                queryset = queryset.filter(**kwquery_1)
+
+        return queryset
+
+    # Developer Interface methods
+    def get_valid_fields(self, queryset, view, context, request):
+        # A query is executed here to get the possible categories
+        category_class = self.get_category_class(view, request)
+        if hasattr(view, 'category_field'):
+            self.category_field = self.get_category_field(view, request)
+
+        assert category_class is not None, (
+            f"{view.__class__.__name__} should include a `category_class`"
+            "attribute"
+        )
+
+        valid_fields = category_class.objects.all()
+
+        if len(valid_fields):
+            valid_fields = [
+                (item.slug, item.__str__()) for item in valid_fields
+            ]
+
+            return valid_fields
+        else:
+            return []
+
+    def get_current(self, request, queryset, view):
+        params = request.query_params.get(self.category_field)
+        if params:
+            fields = [param.strip() for param in params.split(',')]
+            return fields[0]
+        else:
+            return None
+
+    def get_template_context(self, request, queryset, view):
+        current = self.get_current(request, queryset, view)
+        options = []
+        context = {
+            'request': request,
+            'current': current,
+            'param': self.category_field,
+        }
+        valid_fields = self.get_valid_fields(queryset, view, context, request)
+        for key, label in valid_fields:
+            options.append((key, '%s' % (label)))
+        context['options'] = options
+        return context
+
+    def to_html(self, request, queryset, view):
+        template = loader.get_template(self.template)
+        context = self.get_template_context(request, queryset, view)
+        return template.render(context)

starfields_drf_generics/libfilters/facet.py

@@ -0,0 +1,170 @@
+from rest_framework.filters import BaseFilterBackend
+from django.template import loader
+from django.db.models import Q
+
+
+class FacetFilter(BaseFilterBackend):
+    """
+    This filter requires CategoryFilter to be ran before it. It assigns the
+    view.facets which includes all the facets applicable to the current
+    category.
+    """
+    template = 'filters/facets.html'
+
+    def get_facet_class(self, view, request):
+        return getattr(view, 'facet_class', None)
+
+    def get_facet_tag_class(self, view, request):
+        return getattr(view, 'facet_tag_class', None)
+
+    def get_facet_tag_field(self, view, request):
+        return getattr(view, 'facet_tag_field', None)
+
+    def assign_view_facets(self, request, view):
+        if not hasattr(view, 'facets'):
+            if hasattr(view, 'facet_class'):
+                self.facet_class = self.get_facet_class(view, request)
+
+            assert self.facet_class is not None, (
+                f"{view.__class__.__name__} should include a `facet_class`"
+                "attribute"
+            )
+
+            if view.category:
+                if view.category.tn_ancestors_pks:
+                    ancestor_ids = view.category.tn_ancestors_pks.split(',')
+                    view.facets = self.facet_class.objects.filter(
+                        Q(category__id=view.category.id) | Q(
+                            category__id__in=ancestor_ids)
+                        ).prefetch_related('facet_tags')
+                else:
+                    view.facets = self.facet_class.objects.filter(
+                        category__id=view.category.id).prefetch_related(
+                            'facet_tags')
+            else:
+                view.facets = self.facet_class.objects.filter(
+                    category__tn_level=1).prefetch_related(
+                    'facet_tags')
+
+    def get_filters_dict(self, request, view):
+        """
+        Custom method that returns the filters exclusive to this filter in a
+        dict. For caching purposes.
+        """
+        if hasattr(view, 'facet_class'):
+            self.facet_class = self.get_facet_class(view, request)
+
+        assert self.facet_class is not None, (
+            f"{view.__class__.__name__} should include a `facet_class`"
+            "attribute"
+        )
+
+        self.assign_view_facets(request, view)
+
+        filters_dict = {}
+        if view.facets:
+            for facet in view.facets:
+                if facet.slug in request.query_params.keys():
+                    filters_dict[facet.slug] = set(
+                        request.query_params[facet.slug].split(','))
+                else:
+                    filters_dict[facet.slug] = set({})
+
+        # Append the facets object and the tags dict in the view for later
+        # reference
+        view.tags = filters_dict
+
+        return filters_dict
+
+    def filter_queryset(self, request, queryset, view):
+        if hasattr(view, 'facet_tag_class'):
+            self.facet_tag_class = self.get_facet_tag_class(view, request)
+
+        assert self.facet_tag_class is not None, (
+            f"{view.__class__.__name__} should include a `facet_tag_class`"
+            "attribute"
+        )
+
+        if hasattr(view, 'facet_tag_field'):
+            self.facet_tag_field = self.get_facet_tag_field(view, request)
+
+        assert self.facet_tag_field is not None, (
+            f"{view.__class__.__name__} should include a `facet_tag_field`"
+            "attribute"
+        )
+
+        self.assign_view_facets(request, view)
+
+        if view.facets:
+            for facet in view.facets:
+                if facet.slug in request.query_params.keys():
+                    tag_filterlist = request.query_params.get(facet.slug)
+                    if tag_filterlist == '':
+                        # If the tag filterlist is empty then we're not
+                        # filtering against it, it's like having all the tags
+                        # of the facet selected
+                        pass
+                    else:
+                        kwquery = {}
+                        key = self.facet_tag_field+'__slug__in'
+                        kwquery[key] = tag_filterlist.replace(' ', '').split(
+                            ',')
+                        queryset = queryset.filter(**kwquery)
+
+        return queryset
+
+    # Developer Interface methods
+    def get_template_context(self, request, queryset, view):
+        # Does aggressive database querying to get the necessary facets and
+        # facettags, but this is only for the developer interface so its fine
+        if hasattr(view, 'facet_class'):
+            self.facet_class = self.get_facet_class(view, request)
+
+        assert self.facet_class is not None, (
+            f"{view.__class__.__name__} should include a `facet_class`"
+            "attribute"
+        )
+
+        if hasattr(view, 'facet_tag_class'):
+            self.facet_tag_class = self.get_facet_tag_class(view, request)
+
+        assert self.facet_tag_class is not None, (
+            f"{view.__class__.__name__} should include a `facet_tag_class`"
+            "attribute"
+        )
+
+        # Find the current choices
+        current = []
+        facet_slugs = []
+        if view.facets:
+            for facet in view.facets:
+                facet_slugs.append(facet.slug)
+                if facet.slug in request.query_params.keys():
+                    current.append(request.query_params.get(facet.slug))
+
+        facet_slug_names = {}
+        options = {}
+        context = {
+            'request': request,
+            'current': current,
+            'facet_slugs': facet_slugs,
+        }
+        if view.facets:
+            for facet in view.facets:
+                facet_tag_instances = self.facet_tag_class.objects.filter(
+                    facet__slug=facet.slug)
+                options[facet.slug] = [(facet_tag.slug, facet_tag.name) for
+                                       facet_tag in facet_tag_instances]
+                facet_slug_names[facet.slug] = facet.name
+            context['facet_slug_names'] = facet_slug_names
+            context['options'] = options
+        else:
+            context['facet_slug_names'] = {}
+            context['options'] = {}
+
+        return context
+
+    def to_html(self, request, queryset, view):
+        template = loader.get_template(self.template)
+        context = self.get_template_context(request, queryset, view)
+        return template.render(context)

starfields_drf_generics/libfilters/lessthanorequal.py

@@ -0,0 +1,38 @@
+from rest_framework.filters import BaseFilterBackend
+
+
+class LessThanOrEqualFilter(BaseFilterBackend):
+    def get_less_than_field(self, view, request):
+        return getattr(view, 'less_than_field', None)
+
+    def get_filters_dict(self, request, view):
+        """
+        Custom method that returns the filters exclusive to this filter in a
+        dict. For caching purposes.
+        """
+        less_than_field = self.get_less_than_field(view, request)
+
+        assert less_than_field is not None, (
+            f"{view.__class__.__name__} should include a `less_than_field`"
+            "attribute"
+        )
+
+        filters_dict = {}
+        if less_than_field+'_max' in request.query_params.keys():
+            field_value = request.query_params.get(less_than_field+'_max')
+            filters_dict[less_than_field+'_max'] = [field_value]
+        else:
+            filters_dict[less_than_field+'_max'] = []
+        return filters_dict
+
+    def filter_queryset(self, request, queryset, view):
+        # Return the correctly filtered queryset but also assign the filter
+        # dict to create the unique url for the cache
+        less_than_field = self.get_less_than_field(view, request)
+        if less_than_field+'_max' in request.query_params.keys():
+            kwquery = {}
+            field_value = request.query_params.get(less_than_field+'_max')
+            kwquery[less_than_field+'__lte'] = field_value
+            return queryset.filter(**kwquery)
+        else:
+            return queryset

starfields_drf_generics/libfilters/morethanorequal.py

@@ -0,0 +1,39 @@
+from rest_framework.filters import BaseFilterBackend
+
+
+class MoreThanOrEqualFilter(BaseFilterBackend):
+    def get_more_than_field(self, view, request):
+        return getattr(view, 'more_than_field', None)
+
+    def get_filters_dict(self, request, view):
+        """
+        Custom method that returns the filters exclusive to this filter in a
+        dict. For caching purposes.
+        """
+        more_than_field = self.get_more_than_field(view, request)
+
+        assert more_than_field is not None, (
+            f"{view.__class__.__name__} should include a `more_than_field`"
+            "attribute"
+        )
+
+        filters_dict = {}
+        if more_than_field+'_min' in request.query_params.keys():
+            field_value = request.query_params.get(more_than_field+'_min')
+            filters_dict[more_than_field+'_min'] = [field_value]
+        else:
+            filters_dict[more_than_field+'_min'] = []
+        return filters_dict
+
+    def filter_queryset(self, request, queryset, view):
+        """
+        Return the correctly filtered queryset
+        """
+        more_than_field = self.get_more_than_field(view, request)
+        if more_than_field+'_min' in request.query_params.keys():
+            kwquery = {}
+            kwquery[more_than_field+'__gte'] = request.query_params.get(
+                more_than_field+'_min')
+            return queryset.filter(**kwquery)
+        else:
+            return queryset

starfields_drf_generics/libfilters/nodetreebranch.py

@@ -0,0 +1,68 @@
+from rest_framework.settings import api_settings
+from rest_framework.filters import BaseFilterBackend
+from django.db.models import Q
+
+
+class TreeNodeBranchFilter(BaseFilterBackend):
+    def get_descendants_of_field(self, view, request):
+        return getattr(view, 'descendants_of_field', None)
+
+    def get_depth_field(self, view, request):
+        return getattr(view, 'depth_field', None)
+
+    def get_filters_dict(self, request, view):
+        """
+        Custom method that returns the filters exclusive to this filter in a
+        dict. For caching purposes.
+        """
+        descendants_of_field = self.get_descendants_of_field(view, request)
+
+        assert descendants_of_field is not None, (
+            "{view.__class__.__name__} should include a "
+            f"`descendants_of_field` attribute"
+        )
+
+        depth_field = self.get_depth_field(view, request)
+
+        assert depth_field is not None, (
+            "{view.__class__.__name__} should include a "
+            f"`depth_field` attribute"
+        )
+
+        filters_dict = {}
+        if descendants_of_field in request.query_params.keys():
+            field_value = request.query_params.get(descendants_of_field)
+            filters_dict[descendants_of_field] = [field_value]
+        else:
+            filters_dict[descendants_of_field] = []
+
+        if depth_field in request.query_params.keys():
+            field_value = request.query_params.get(depth_field)
+            filters_dict[depth_field] = [field_value]
+        else:
+            filters_dict[depth_field] = [4]
+
+        return filters_dict
+
+    def filter_queryset(self, request, queryset, view):
+        # Return the correctly filtered queryset but also assign the filter
+        # dict to create the unique url for the cache
+        descendants_of_field = self.get_descendants_of_field(view, request)
+        depth_field = self.get_depth_field(view, request)
+
+        if descendants_of_field in request.query_params.keys():
+            field_value = request.query_params.get(descendants_of_field)
+            # Instead of doing two queries to get the descendants through the
+            # object a single more complex queryset
+            queryset = queryset.filter(
+                Q(tn_ancestors_pks=field_value) |
+                Q(tn_ancestors_pks__contains=","+field_value) |
+                Q(tn_ancestors_pks__contains=field_value+","))
+
+        if depth_field in request.query_params.keys():
+            field_value = request.query_params.get(depth_field)
+            queryset = queryset.filter(tn_level__lte=field_value)
+        else:
+            queryset = queryset.filter(tn_level__lte=4)
+
+        return queryset

starfields_drf_generics/libfilters/ordering.py

@@ -0,0 +1,180 @@
+from rest_framework.settings import api_settings
+from rest_framework.filters import BaseFilterBackend
+from django.template import loader
+from django.utils.translation import gettext_lazy as _
+
+
+class OrderingFilter(BaseFilterBackend):
+    # The URL query parameter used for the ordering.
+    ordering_param = api_settings.ORDERING_PARAM
+    ordering_fields = None
+    ordering_title = _('Ordering')
+    ordering_description = _('Which field to use when ordering the results.')
+    template = 'rest_framework/filters/ordering.html'
+
+    def get_ordering(self, request, queryset, view):
+        """
+        Ordering is set by a comma delimited ?ordering=... query parameter.
+
+        The `ordering` query parameter can be overridden by setting
+        the `ordering_param` value on the OrderingFilter or by
+        specifying an `ORDERING_PARAM` value in the API settings.
+        """
+        params = request.query_params.get(self.ordering_param)
+        if params:
+            fields = [param.strip() for param in params.split(',')]
+            ordering = self.remove_invalid_fields(queryset,
+                                                  fields,
+                                                  view,
+                                                  request)
+            if ordering:
+                return ordering
+
+        # No ordering was included, or all the ordering fields were invalid
+        return self.get_default_ordering(view)
+
+    def get_default_ordering(self, view):
+        ordering = getattr(view, 'ordering', None)
+        if isinstance(ordering, str):
+            return (ordering,)
+        return ordering
+
+    def get_default_valid_fields(self, queryset, view, context={}):
+        # If `ordering_fields` is not specified, then we determine a default
+        # based on the serializer class, if one exists on the view.
+        if hasattr(view, 'get_serializer_class'):
+            try:
+                serializer_class = view.get_serializer_class()
+            except AssertionError:
+                # Raised by the default implementation if
+                # no serializer_class was found
+                serializer_class = None
+        else:
+            serializer_class = getattr(view, 'serializer_class', None)
+
+        if serializer_class is None:
+            msg = (
+                "Cannot use %s on a view which does not have either a "
+                "'serializer_class', an overriding 'get_serializer_class' "
+                "or 'ordering_fields' attribute."
+            )
+            raise ImproperlyConfigured(msg % self.__class__.__name__)
+
+        model_class = queryset.model
+        model_property_names = [
+            # 'pk' is a property added in Django's Model class, however it is valid for ordering.
+            attr for attr in dir(model_class) if isinstance(getattr(model_class, attr), property) and attr != 'pk'
+        ]
+
+        return [
+            (field.source.replace('.', '__') or field_name, field.label)
+            for field_name, field in serializer_class(context=context).fields.items()
+            if (
+                not getattr(field, 'write_only', False) and
+                not field.source == '*' and
+                field.source not in model_property_names
+            )
+        ]
+
+    def get_valid_fields(self, queryset, view, context={}):
+        valid_fields = getattr(view, 'ordering_fields', self.ordering_fields)
+
+        if valid_fields is None:
+            # Default to allowing filtering on serializer fields
+            return self.get_default_valid_fields(queryset, view, context)
+
+        elif valid_fields == '__all__':
+            # View explicitly allows filtering on any model field
+            valid_fields = [
+                (field.name, field.verbose_name) for field in queryset.model._meta.fields
+            ]
+            valid_fields += [
+                (key, key.title().split('__'))
+                for key in queryset.query.annotations
+            ]
+        else:
+            valid_fields = [
+                (item, item) if isinstance(item, str) else item
+                for item in valid_fields
+            ]
+
+        return valid_fields
+
+    def remove_invalid_fields(self, queryset, fields, view, request):
+        valid_fields = [item[0] for item in self.get_valid_fields(queryset, view, {'request': request})]
+
+        def term_valid(term):
+            if term.startswith("-"):
+                term = term[1:]
+            return term in valid_fields
+
+        return [term for term in fields if term_valid(term)]
+
+    def get_filters_dict(self, request, view):
+        """
+        Custom method that returns the filters exclusive to this filter in a dict. For caching purposes.
+        """
+        self.filters_dict = {}
+        if 'ordering' in request.query_params.keys():
+            slug_term = request.query_params.get('ordering')
+            self.filters_dict['ordering'] = [slug_term]
+        else:
+            self.filters_dict['ordering'] = [view.ordering_fields[0]]
+
+        return self.filters_dict
+
+    def filter_queryset(self, request, queryset, view):
+        ordering = self.get_ordering(request, queryset, view)
+
+        if ordering:
+            return queryset.order_by(*ordering)
+
+        return queryset
+
+    def get_template_context(self, request, queryset, view):
+        current = self.get_ordering(request, queryset, view)
+        current = None if not current else current[0]
+        options = []
+        context = {
+            'request': request,
+            'current': current,
+            'param': self.ordering_param,
+        }
+        for key, label in self.get_valid_fields(queryset, view, context):
+            options.append((key, '%s - %s' % (label, _('ascending'))))
+            options.append(('-' + key, '%s - %s' % (label, _('descending'))))
+        context['options'] = options
+        return context
+
+    def to_html(self, request, queryset, view):
+        template = loader.get_template(self.template)
+        context = self.get_template_context(request, queryset, view)
+        return template.render(context)
+
+    def get_schema_fields(self, view):
+        assert coreapi is not None, 'coreapi must be installed to use `get_schema_fields()`'
+        assert coreschema is not None, 'coreschema must be installed to use `get_schema_fields()`'
+        return [
+            coreapi.Field(
+                name=self.ordering_param,
+                required=False,
+                location='query',
+                schema=coreschema.String(
+                    title=force_str(self.ordering_title),
+                    description=force_str(self.ordering_description)
+                )
+            )
+        ]
+
+    def get_schema_operation_parameters(self, view):
+        return [
+            {
+                'name': self.ordering_param,
+                'required': False,
+                'in': 'query',
+                'description': force_str(self.ordering_description),
+                'schema': {
+                    'type': 'string',
+                },
+            },
+        ]

starfields_drf_generics/libfilters/trigramsearch.py

@@ -0,0 +1,247 @@
+from django.utils.text import smart_split
+from django.core.exceptions import FieldError, FieldDoesNotExist
+import operator
+from functools import reduce
+from rest_framework.filters import BaseFilterBackend
+from django.template import loader
+from django.utils.translation import gettext_lazy as _
+from django.db import models
+from django.db.models import Q
+from rest_framework.fields import CharField
+from django.db.models.constants import LOOKUP_SEP
+from django.db.models.functions import Concat
+
+
+def calculate_threshold(query, min_threshold, max_threshold):
+    query_threshold = len(query)/300
+    if query_threshold < min_threshold:
+        return min_threshold
+    if max_threshold < query_threshold:
+        return max_threshold
+    return query_threshold
+
+
+def search_smart_split(search_terms):
+    """
+    Generator that first splits string by spaces, leaving quoted phrases
+    together, then it splits non-quoted phrases by commas.
+    """
+    split_terms = []
+    for term in smart_split(search_terms):
+        # trim commas to avoid bad matching for quoted phrases
+        term = term.strip(',')
+        if term.startswith(('"', "'")) and term[0] == term[-1]:
+            # quoted phrases are kept together without any other split
+            split_terms.append(unescape_string_literal(term))
+        else:
+            # non-quoted tokens are split by comma, keeping only non-empty ones
+            for sub_term in term.split(','):
+                if sub_term:
+                    split_terms.append(sub_term.strip())
+    return split_terms
+
+
+class TrigramSearchFilter(BaseFilterBackend):
+    # The URL query parameter used for the search.
+    search_param = 'search'
+    template = 'rest_framework/filters/search.html'
+    search_title = _('Search')
+    search_description = _('A search string to perform trigram similarity'
+                           'based searching with.')
+    lookup_prefixes = {
+        '^': 'istartswith',
+        '=': 'iexact',
+        '@': 'search',
+        '$': 'iregex',
+    }
+
+    def get_filters_dict(self, request, view):
+        """
+        Custom method that returns the filters exclusive to this filter in a
+        dict. For caching purposes.
+        """
+        self.filters_dict = {}
+        if 'search' in request.query_params.keys():
+            slug_term = request.query_params.get('search')
+            self.filters_dict['search'] = [slug_term]
+        else:
+            self.filters_dict['search'] = []
+
+        return self.filters_dict
+
+    def get_search_fields(self, view, request):
+        """
+        Search fields are obtained from the view, but the request is always
+        passed to this method. Sub-classes can override this method to
+        dynamically change the search fields based on request content.
+        """
+        return getattr(view, 'search_fields', None)
+
+    def get_search_query(self, request):
+        """
+        Search terms are set by a ?search=... query parameter,
+        and may be whitespace delimited.
+        """
+        value = request.query_params.get(self.search_param, '')
+        field = CharField(trim_whitespace=False, allow_blank=True)
+        cleaned_value = field.run_validation(value)
+        return cleaned_value
+
+    def construct_search(self, field_name, queryset):
+        """
+        For the sqlite search
+        """
+        lookup = self.lookup_prefixes.get(field_name[0])
+        if lookup:
+            field_name = field_name[1:]
+        else:
+            # Use field_name if it includes a lookup.
+            opts = queryset.model._meta
+            lookup_fields = field_name.split(LOOKUP_SEP)
+            # Go through the fields, following all relations.
+            prev_field = None
+            for path_part in lookup_fields:
+                if path_part == "pk":
+                    path_part = opts.pk.name
+                try:
+                    field = opts.get_field(path_part)
+                except FieldDoesNotExist:
+                    # Use valid query lookups.
+                    if prev_field and prev_field.get_lookup(path_part):
+                        return field_name
+                else:
+                    prev_field = field
+                    if hasattr(field, "path_infos"):
+                        # Update opts to follow the relation.
+                        opts = field.path_infos[-1].to_opts
+                    # django < 4.1
+                    elif hasattr(field, 'get_path_info'):
+                        # Update opts to follow the relation.
+                        opts = field.get_path_info()[-1].to_opts
+            # Otherwise, use the field with icontains.
+            lookup = 'icontains'
+        return LOOKUP_SEP.join([field_name, lookup])
+
+    def must_call_distinct(self, queryset, search_fields):
+        """
+        Return True if 'distinct()' should be used to query the given lookups.
+        """
+        for search_field in search_fields:
+            opts = queryset.model._meta
+            if search_field[0] in self.lookup_prefixes:
+                search_field = search_field[1:]
+            # Annotated fields do not need to be distinct
+            if isinstance(queryset, models.QuerySet) and search_field in queryset.query.annotations:
+                continue
+            parts = search_field.split(LOOKUP_SEP)
+            for part in parts:
+                field = opts.get_field(part)
+                if hasattr(field, 'get_path_info'):
+                    # This field is a relation, update opts to follow the relation
+                    path_info = field.get_path_info()
+                    opts = path_info[-1].to_opts
+                    if any(path.m2m for path in path_info):
+                        # This field is a m2m relation so we know we need to call distinct
+                        return True
+                else:
+                    # This field has a custom __ query transform but is not a relational field.
+                    break
+        return False
+
+    def filter_queryset(self, request, queryset, view):
+        search_fields = self.get_search_fields(view, request)
+
+        assert search_fields is not None, (
+            f"{view.__class__.__name__} should include a `search_fields`"
+            "attribute"
+        )
+
+        query = self.get_search_query(request)
+
+        if not query:
+            return queryset
+
+        try:
+            # Attempt postgresql's full text search
+            from django.contrib.postgres.search import TrigramStrictWordSimilarity
+            threshold = calculate_threshold(query, 0.02, 0.12)
+            queryset = queryset.annotate(
+                            search_field=Concat(
+                                *search_fields,
+                                output_field=CharField()
+                                )).annotate(
+                        similarity=TrigramStrictWordSimilarity(
+                            'search_field', query)
+                    ).filter(similarity__gt=threshold)
+
+        # NOTE a weird FieldError is raised on sqlite
+        except (ImportError, FieldError):
+            # Perform very simple sqlite compatible search
+            search_terms = search_smart_split(query)
+
+            orm_lookups = [
+                self.construct_search(str(search_field), queryset)
+                for search_field in search_fields
+            ]
+
+            base = queryset
+            # generator which for each term builds the corresponding search
+            conditions = (
+                reduce(
+                    operator.or_,
+                    (models.Q(**{orm_lookup: term}) for orm_lookup in orm_lookups)
+                ) for term in search_terms
+            )
+            queryset = queryset.filter(reduce(operator.and_, conditions))
+
+        # Remove duplicates from results, if necessary
+        if self.must_call_distinct(queryset, search_fields):
+            # inspired by django.contrib.admin
+            # this is more accurate than .distinct form M2M relationship
+            # also is cross-database
+            queryset = queryset.filter(pk=models.OuterRef('pk'))
+            queryset = base.filter(models.Exists(queryset))
+
+        return queryset
+
+    def to_html(self, request, queryset, view):
+        if not getattr(view, 'search_fields', None):
+            return ''
+
+        term = request.query_params.get(self.search_param, '')
+        term = term[0] if term else ''
+        context = {
+            'param': self.search_param,
+            'term': term
+        }
+        template = loader.get_template(self.template)
+        return template.render(context)
+
+    def get_schema_fields(self, view):
+        assert coreapi is not None, 'coreapi must be installed to use `get_schema_fields()`'
+        assert coreschema is not None, 'coreschema must be installed to use `get_schema_fields()`'
+        return [
+            coreapi.Field(
+                name=self.search_param,
+                required=False,
+                location='query',
+                schema=coreschema.String(
+                    title=force_str(self.search_title),
+                    description=force_str(self.search_description)
+                )
+            )
+        ]
+
+    def get_schema_operation_parameters(self, view):
+        return [
+            {
+                'name': self.search_param,
+                'required': False,
+                'in': 'query',
+                'description': force_str(self.search_description),
+                'schema': {
+                    'type': 'string',
+                },
+            },
+        ]
+

starfields_drf_generics/mixins.py

@@ -8,15 +8,19 @@ from rest_framework import status
 from rest_framework.response import Response
 from rest_framework.settings import api_settings
 from rest_framework import mixins
-from starfields_drf_generics.cache_mixins import CacheGetMixin, CacheSetMixin, CacheDeleteMixin
+from starfields_drf_generics.cache_mixins import (
+    CacheGetMixin, CacheSetMixin, CacheDeleteMixin)
 
 
 # Mixin classes to be included in the generic classes
 class CachedCreateModelMixin(CacheDeleteMixin, mixins.CreateModelMixin):
     """
-    A slightly modified version of rest_framework.mixins.CreateModelMixin that handles cache deletions.
+    A slightly modified version of rest_framework.mixins.CreateModelMixin
+    that handles cache deletions.
     """
+
     def create(self, request, *args, **kwargs):
+        " Creates the entry in the request "
         # Go on with the creation as normal
         serializer = self.get_serializer(data=request.data)
         serializer.is_valid(raise_exception=True)
@@ -25,22 +29,28 @@ class CachedCreateModelMixin(CacheDeleteMixin, mixins.CreateModelMixin):
 
         # Delete the cache
         self.delete_cache(request)
-        return Response(serializer.data, status=status.HTTP_201_CREATED, headers=headers)
+        return Response(serializer.data, status=status.HTTP_201_CREATED,
+                        headers=headers)
 
 
 class CachedRetrieveModelMixin(CacheGetMixin, CacheSetMixin):
     """
-    A slightly modified version of rest_framework.mixins.RetrieveModelMixin that handles cache attempts.
-    mixins.RetrieveModelMixin only has the retrieve method so it doesn't stand to inherit anything from it.
+    A slightly modified version of rest_framework.mixins.RetrieveModelMixin
+    that handles cache attempts.
+    mixins.RetrieveModelMixin only has the retrieve method so it doesn't stand
+    to inherit anything from it.
     """
+
     def retrieve(self, request, *args, **kwargs):
+        " Retrieves the entry in the request "
         # Attempt to get the request from the cache
         cache_attempt = self.get_cache(request)
 
         if cache_attempt:
             return Response(cache_attempt)
-        else:
-            # The cache get attempt failed so we have to get the results from the database
+
+        # The cache get attempt failed so we have to get the results from
+        # the database
         instance = self.get_object()
 
         serializer = self.get_serializer(instance)
@@ -52,12 +62,16 @@ class CachedRetrieveModelMixin(CacheGetMixin, CacheSetMixin):
 
 class CachedUpdateModelMixin(CacheDeleteMixin, mixins.UpdateModelMixin):
     """
-    A slightly modified version of rest_framework.mixins.UpdateModelMixin that handles cache deletes.
+    A slightly modified version of rest_framework.mixins.UpdateModelMixin that
+    handles cache deletes.
     """
+
     def update(self, request, *args, **kwargs):
+        " Updates the entry in the request "
         partial = kwargs.pop('partial', False)
         instance = self.get_object()
-        serializer = self.get_serializer(instance, data=request.data, partial=partial)
+        serializer = self.get_serializer(instance, data=request.data,
+                                         partial=partial)
         serializer.is_valid(raise_exception=True)
         self.perform_update(serializer)
 
@@ -74,9 +88,12 @@ class CachedUpdateModelMixin(CacheDeleteMixin, mixins.UpdateModelMixin):
 
 class CachedDestroyModelMixin(CacheDeleteMixin, mixins.DestroyModelMixin):
     """
-    A slightly modified version of rest_framework.mixins.DestroyModelMixin that handles cache deletes.
+    A slightly modified version of rest_framework.mixins.DestroyModelMixin
+    that handles cache deletes.
     """
+
     def destroy(self, request, *args, **kwargs):
+        " Deletes the entry in the request "
         instance = self.get_object()
         self.perform_destroy(instance)
 
@@ -91,7 +108,9 @@ class CachedListCreateModelMixin(CacheDeleteMixin):
     """
     A fully custom mixin that handles mutiple instance cration.
     """
-    def list_create(self, request, *args, **kwargs):
+
+    def list_create(self, request, **kwargs):
+        " Creates the list of entries in the request "
         # Go on with the creation as normal
         serializer = self.get_serializer(data=request.data, many=True)
         serializer.is_valid(raise_exception=True)
@@ -100,12 +119,31 @@ class CachedListCreateModelMixin(CacheDeleteMixin):
 
         # Delete the cache
         self.delete_cache(request)
-        return Response(serializer.data, status=status.HTTP_201_CREATED, headers=headers)
+        return Response(serializer.data, status=status.HTTP_201_CREATED,
+                        headers=headers)
 
     def perform_create(self, serializer):
-        serializer.save()
+        """
+        Uses serializer.create instead of serializer.save to avoid making a
+        query. We save the returned instance list to the serializer in order to
+        be used as serializer.data during rendering
+        """
+        assert hasattr(serializer, 'create'), (
+            f'Cannot call .create() on serializer {serializer.__class__} as'
+            ' no such attribute exists.'
+        )
+        validated_data = serializer.validated_data
+        instance_list = serializer.create(validated_data)
+        # Check whatever you can
+        assert hasattr(instance_list, '__iter__'), (
+            'Method .create() on serializer on serializer '
+            f'{serializer.__class__} should return a list of serializable'
+            ' model instances.'
+        )
+        serializer.instance = instance_list
 
     def get_success_headers(self, data):
+        " Returns extra success headers "
         try:
             return {'Location': str(data[api_settings.URL_FIELD_NAME])}
         except (TypeError, KeyError):
@@ -114,17 +152,22 @@ class CachedListCreateModelMixin(CacheDeleteMixin):
 
 class CachedListRetrieveModelMixin(CacheGetMixin, CacheSetMixin):
     """
-    A slightly modified version of rest_framework.mixins.ListModelMixin that handles cache saves.
-    mixins.ListModelMixin only has the list method so it doesn't stand to inherit anything from it.
+    A slightly modified version of rest_framework.mixins.ListModelMixin that
+    handles cache saves.
+    mixins.ListModelMixin only has the list method so it doesn't stand to
+    inherit anything from it.
     """
-    def list(self, request, *args, **kwargs):
+
+    def list(self, request, **kwargs):
+        " Retrieves the listing of entries "
         # Attempt to get the request from the cache
         cache_attempt = self.get_cache(request)
 
         if cache_attempt:
             return Response(cache_attempt)
-        else:
-            # The cache get attempt failed so we have to get the results from the database
+
+        # The cache get attempt failed so we have to get the results from
+        # the database
         queryset = self.filter_queryset(self.get_queryset())
 
         if self.paged:
@@ -147,12 +190,13 @@ class CachedListUpdateModelMixin(CacheDeleteMixin):
     """
     A fully custom mixin that handles mutiple instance updates.
     """
-    def list_update(self, request, *args, **kwargs):
+
+    def list_update(self, request, **kwargs):
+        " Updates the list of entries in the request "
         partial = kwargs.pop('partial', False)
 
-        queryset = self.filter_queryset(self.get_queryset())
-        
-        serializer = self.get_serializer(queryset, data=request.data, partial=partial, many=True)
+        serializer = self.get_serializer(data=request.data,
+                                         partial=partial, many=True)
         serializer.is_valid(raise_exception=True)
         self.perform_update(serializer)
 
@@ -162,9 +206,27 @@ class CachedListUpdateModelMixin(CacheDeleteMixin):
         return Response(serializer.data)
 
     def perform_update(self, serializer):
-        serializer.save()
+        """
+        Uses serializer.update instead of serializer.save to avoid making a
+        query. We save the returned instance list to the serializer in order to
+        be used as serializer.data during rendering
+        """
+        assert hasattr(serializer, 'update'), (
+            f'Cannot call .update() on serializer {serializer.__class__} as'
+            ' no such attribute exists.'
+        )
+        validated_data = serializer.validated_data
+        instance_list = serializer.update(None, validated_data)
+        # Check whatever you can
+        assert hasattr(instance_list, '__iter__'), (
+            'Method .update() on serializer on serializer '
+            f'{serializer.__class__} should return a list of serializable '
+            ' model instances.'
+        )
+        serializer.instance = instance_list
 
     def list_partial_update(self, request, *args, **kwargs):
+        " Needs to be called on partial updates "
         kwargs['partial'] = True
         return self.list_update(request, *args, **kwargs)
 
@@ -173,39 +235,25 @@ class CachedListDestroyModelMixin(CacheDeleteMixin):
     """
     A fully custom mixin that handles mutiple instance deletions.
     """
-    def list_destroy(self, request, *args, **kwargs):
+
+    def list_destroy(self, request, **kwargs):
+        " Deletes the list of entries in the request "
         # Go on with the validation as normal
         serializer = self.get_serializer(data=request.data, many=True)
         serializer.is_valid(raise_exception=True)
-        validated_data = serializer.validated_data
 
-        # TODO does this new stuff work even? need to check on the frontend
-        serializer.delete(validated_data)
-        
-#        for instance in self.get_objects():
-#            if instance is not None:
-#                self.perform_destroy(instance)
+        self.perform_destroy(serializer)
 
         # Delete the related caches
         self.delete_cache(request)
 
         return Response(status=status.HTTP_204_NO_CONTENT)
 
-    #def perform_destroy(self, instance):
-    #    instance.delete()
-
-    #def get_objects(self):
-    #    """
-    #    The custom list version of get_object that retrieves one instance from the #database. It yields model instances with each call.
-    #    """
-    #    queryset = self.filter_queryset(self.get_queryset())
-    #    
-    #    if len(queryset):
-    #        for obj in queryset.all():
-
-    #            # May raise a permission denied
-    #            self.check_object_permissions(self.request, obj)
-
-        #        yield obj
-            
-        #yield None
+    def perform_destroy(self, serializer):
+        " Custom generic destroy hook "
+        assert hasattr(serializer, 'destroy'), (
+            f'Cannot call .destroy() on serializer {serializer.__class__} as'
+            ' no such attribute exists.'
+        )
+        validated_data = serializer.validated_data
+        serializer.destroy(validated_data)

starfields_drf_generics/parsers.py

@@ -0,0 +1,73 @@
+from rest_framework.parsers import BaseParser, DataAndFiles
+from django.http.multipartparser import MultiPartParserError
+from rest_framework.exceptions import ParseError
+from django.http.multipartparser import \
+    MultiPartParser as DjangoMultiPartParser
+from django.conf import settings
+import json
+
+
+class NestedJsonMultiPartParser(BaseParser):
+    """
+    Parser for multipart form data, which may include file data.
+    """
+    media_type = 'multipart/form-data'
+
+    def parse(self, stream, media_type=None, parser_context=None):
+        """
+        Parses the incoming bytestream as a multipart encoded form,
+        and returns a DataAndFiles object.
+        The main difference with the parser from rest_framework.parsers
+        is that it attempts to parse nested strings as json to fit with
+        better with json payloads. I also ensure that a single file is
+        passed per field instead of a list which was erroring out
+        FieldFile.to_internal_value.
+
+        `.data` will be a dict containing all the form parameters.
+        `.files` will be a dict containing all the form files.
+        """
+        parser_context = parser_context or {}
+        request = parser_context['request']
+        encoding = parser_context.get('encoding', settings.DEFAULT_CHARSET)
+        meta = request.META.copy()
+        meta['CONTENT_TYPE'] = media_type
+        upload_handlers = request.upload_handlers
+
+        try:
+            parser = DjangoMultiPartParser(meta,
+                                           stream,
+                                           upload_handlers,
+                                           encoding)
+            data, files = parser.parse()
+
+            # Attempt to parse the multipart fields as json, this is not
+            # demanding since multiparts exist exclusively for file uploads
+            # which is much more demanding
+            data_dict = {}
+            for key in data.keys():
+                values = data[key]
+                try:
+                    data_dict[key] = json.loads(values)
+                except Exception as e:
+                    data_dict[key] = values
+
+            # Make sure the filenames become file names
+            for filename in files.keys():
+                uploaded_file = files[filename]
+                hasfilename = hasattr(uploaded_file, '_name')
+                hasname = hasattr(uploaded_file, 'name')
+                if hasfilename and not hasname:
+                    uploaded_file.name = uploaded_file._name
+
+            # Turn the monstrous MultiValueDict into a normal dict so that
+            # there is only a single uploaded file per field passed
+            files_dict = {}
+            for key in files.keys():
+                uploaded_file = files[key]
+                if isinstance(uploaded_file, list):
+                    uploaded_file = uploaded_file[0]
+                files_dict[key] = uploaded_file
+
+            return DataAndFiles(data_dict, files_dict)
+        except MultiPartParserError as exc:
+            raise ParseError('Multipart form parse error - %s' % str(exc))

starfields_drf_generics/utils.py

@@ -1,18 +1,26 @@
+"""
+    Utility functions for the library
+"""
+
 def sorted_params_string(filters_dict):
     """
-    This function takes a dict and returns it in a sorted form for the url filter, it's primarily used for cache purposes.
+    This function takes a dict and returns it in a sorted form for the url
+    filter, it's primarily used for cache purposes.
     """
     filters_string = ''
     for key in sorted(filters_dict.keys()):
         if filters_string == '':
-            filters_string = f"{key}={','.join(str(val) for val in sorted(filters_dict[key]))}"
+            key_str = ','.join(str(val) for val in sorted(filters_dict[key]))
+            filters_string = f"{key}={key_str}"
         else:
-            filters_string = f"{filters_string}&{key}={','.join(str(val) for val in sorted(filters_dict[key]))}"
+            key_str = ','.join(str(val) for val in sorted(filters_dict[key]))
+            filters_string = f"{filters_string}&{key}={key_str}"
         filters_string = filters_string.strip()
     return filters_string
 
 
 def parse_tags_to_dict(tags):
+    " This function parses a tag string into a dictionary "
     tagdict = {}
     if ':' not in tags:
         tagdict = {}